[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:23 dist-test-slave-vxj2 krb5kdc[24827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:23 dist-test-slave-vxj2 krb5kdc[24827](info): set up 2 sockets
Aug 10 07:24:23 dist-test-slave-vxj2 krb5kdc[24827](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 07:24:24.420274 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:24 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:24.553501 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45219
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:24.640055 24843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:24.640281 24843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:24.640305 24843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:24.641542 24843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:24.641610 24843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:24.641626 24843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:24.641639 24843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:24.641651 24843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:24.643290 24843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:24.643599 24843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:24.643862 24843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:24.649233 24848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:24.649245 24851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:24.649245 24849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:24.649569 24843 server_base.cc:1034] running on GCE node
I20260810 07:24:24.649843 24843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:24.650224 24843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:24.651381 24843 hybrid_clock.cc:648] HybridClock initialized: now 1786346664651374 us; error 116 us; skew 500 ppm
Aug 10 07:24:24 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:24.654120 24843 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:24.654625 24843 webserver.cc:460] Webserver started at http://127.24.60.254:41665/ using document root <none> and password file <none>
I20260810 07:24:24.654873 24843 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:24.654943 24843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:24.655105 24843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:24.656241 24843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "2d5ade23896e48f89e23ba7c4477f7b9"
format_stamp: "Formatted at 2026-08-10 07:24:24 on dist-test-slave-vxj2"
server_key: "4342a8a31def48bc514b4893d2081f40"
server_key_iv: "623c43116646723b59351cb8792f59dd"
server_key_version: "encryptionkey@0"
I20260810 07:24:24.656642 24843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "2d5ade23896e48f89e23ba7c4477f7b9"
format_stamp: "Formatted at 2026-08-10 07:24:24 on dist-test-slave-vxj2"
server_key: "4342a8a31def48bc514b4893d2081f40"
server_key_iv: "623c43116646723b59351cb8792f59dd"
server_key_version: "encryptionkey@0"
I20260810 07:24:24.658195 24843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:24.659031 24859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:24.659240 24843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:24.659338 24843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "2d5ade23896e48f89e23ba7c4477f7b9"
format_stamp: "Formatted at 2026-08-10 07:24:24 on dist-test-slave-vxj2"
server_key: "4342a8a31def48bc514b4893d2081f40"
server_key_iv: "623c43116646723b59351cb8792f59dd"
server_key_version: "encryptionkey@0"
I20260810 07:24:24.659430 24843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:24.679112 24843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:24.681645 24843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:24.681896 24843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:24.686102 24843 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:45219
I20260810 07:24:24.686141 24916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:45219 every 8 connection(s)
I20260810 07:24:24.686547 24843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:24.687530 24917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:24.690008 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 24843
I20260810 07:24:24.690130 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:24.690541 24917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9: Bootstrap starting.
I20260810 07:24:24.690955 24819 external_mini_cluster.cc:1428] Setting key 6968828937c562967b6162b9f822356a
I20260810 07:24:24.691305 24917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:24.691668 24917 log.cc:824] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:24.692721 24917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9: No bootstrap required, opened a new log
I20260810 07:24:24.694646 24917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45219 } }
I20260810 07:24:24.694847 24917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:24.694888 24917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5ade23896e48f89e23ba7c4477f7b9, State: Initialized, Role: FOLLOWER
I20260810 07:24:24.695019 24917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45219 } }
I20260810 07:24:24.695101 24917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:24.695125 24917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:24.695191 24917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:24.696106 24917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45219 } }
I20260810 07:24:24.696210 24917 leader_election.cc:302] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d5ade23896e48f89e23ba7c4477f7b9; no voters:
I20260810 07:24:24.696473 24917 leader_election.cc:288] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:24.696508 24922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:24.696635 24922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5ade23896e48f89e23ba7c4477f7b9, State: Running, Role: LEADER
I20260810 07:24:24.696728 24922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45219 } }
I20260810 07:24:24.696789 24917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:24.697105 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d5ade23896e48f89e23ba7c4477f7b9. Latest consensus state: current_term: 1 leader_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45219 } } }
I20260810 07:24:24.697216 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:24.697115 24924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5ade23896e48f89e23ba7c4477f7b9" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45219 } } }
I20260810 07:24:24.697264 24924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:24.697474 24927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:24.698305 24927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:24.700888 24927 catalog_manager.cc:1347] Generated new cluster ID: f8c1f14b5b5e42118cb529ac4cefcc04
I20260810 07:24:24.700958 24927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:24:24 dist-test-slave-vxj2 krb5kdc[24827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:24.708819 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:24.692035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.24.60.254:45219)
0810 07:24:24.692227 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:24.692236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:24.692462 (+ 226us) server_negotiation.cc:407] Connection header received
0810 07:24:24.694599 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:24.694608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:24.694686 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:24.694872 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:24.696807 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:24.698266 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:24.699993 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:24.700620 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:24.700752 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:24.704647 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:24.704653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:24.704653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:24.704669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:24.706580 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:24.707092 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:24.707097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:24.707097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:24.707213 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:24.707513 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:24.707514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:24.707514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:24.707801 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:24.707998 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:24.708278 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 07:24:24.708441 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260810 07:24:24.710412 24927 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:24.711122 24927 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:24.714593 24927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d5ade23896e48f89e23ba7c4477f7b9: Generated new TSK 0
I20260810 07:24:24.714769 24927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:24.749385 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45219
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:24.840464 24943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:24.840739 24943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:24.840788 24943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:24.842056 24943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:24.842141 24943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:24.842190 24943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:24.843823 24943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:24.844136 24943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:24.844400 24943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:24.850255 24950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:24.850245 24943 server_base.cc:1034] running on GCE node
W20260810 07:24:24.850245 24952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:24.850289 24949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:24.850806 24943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:24.851155 24943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:24.852334 24943 hybrid_clock.cc:648] HybridClock initialized: now 1786346664852314 us; error 45 us; skew 500 ppm
Aug 10 07:24:24 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:24.855314 24943 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:24.855849 24943 webserver.cc:460] Webserver started at http://127.24.60.193:34003/ using document root <none> and password file <none>
I20260810 07:24:24.856097 24943 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:24.856145 24943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:24.856283 24943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:24.857481 24943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "faae9474524c4a138f7e54e544637dc6"
format_stamp: "Formatted at 2026-08-10 07:24:24 on dist-test-slave-vxj2"
server_key: "e3f74aa25785823f6980811927e73b73"
server_key_iv: "3abc822fe5d7a513d9ea99a3a4ee9098"
server_key_version: "encryptionkey@0"
I20260810 07:24:24.857851 24943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "faae9474524c4a138f7e54e544637dc6"
format_stamp: "Formatted at 2026-08-10 07:24:24 on dist-test-slave-vxj2"
server_key: "e3f74aa25785823f6980811927e73b73"
server_key_iv: "3abc822fe5d7a513d9ea99a3a4ee9098"
server_key_version: "encryptionkey@0"
I20260810 07:24:24.859340 24943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:24.860263 24961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:24.860500 24943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:24.860620 24943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "faae9474524c4a138f7e54e544637dc6"
format_stamp: "Formatted at 2026-08-10 07:24:24 on dist-test-slave-vxj2"
server_key: "e3f74aa25785823f6980811927e73b73"
server_key_iv: "3abc822fe5d7a513d9ea99a3a4ee9098"
server_key_version: "encryptionkey@0"
I20260810 07:24:24.860700 24943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:24.878250 24943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:24.881038 24943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:24.881276 24943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:24.881572 24943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:24.881942 24943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:24.882011 24943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:24.882078 24943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:24.882122 24943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:24.889050 24943 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:44027
I20260810 07:24:24.889101 25082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:44027 every 8 connection(s)
I20260810 07:24:24.889537 24943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:24.895812 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 24943
I20260810 07:24:24.895941 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:24.896103 24819 external_mini_cluster.cc:1428] Setting key c9dd60887dafa81543aaab330dcd1159
Aug 10 07:24:24 dist-test-slave-vxj2 krb5kdc[24827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346664, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:24.901473 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:24.890530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:48813 (local address 127.24.60.254:45219)
0810 07:24:24.890604 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:24.890606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:24.890876 (+ 270us) server_negotiation.cc:407] Connection header received
0810 07:24:24.891473 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:24.891475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:24.891490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:24.891552 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:24.892704 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:24.893473 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:24.894555 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:24.894757 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:24.894813 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:24.897725 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:24.897727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:24.897727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:24.897735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:24.899626 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:24.900208 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:24.900209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:24.900209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:24.900265 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:24.900622 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:24.900623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:24.900623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:24.900752 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:24.900891 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:24.901289 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 07:24:24.901404 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:24:24.901785 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:24.890510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45219 (local address 127.24.60.193:48813)
0810 07:24:24.890728 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:24.890771 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:24.891351 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:24.891643 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:24.891647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:24.892370 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:24.892595 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:24.892601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:24.893569 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:24.893570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:24.894473 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:24.894474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:24.894838 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:24.894839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:24.895013 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:24.895566 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:24.895576 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:24.897639 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:24.899748 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:24.899756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:24.899758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:24.900114 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:24.900358 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:24.900359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:24.900360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:24.900525 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:24.900898 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:24.900905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:24.901097 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 07:24:24.901246 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 07:24:24.901474 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260810 07:24:24.902431 25083 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45219
I20260810 07:24:24.902536 25083 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:24.902752 25083 heartbeater.cc:511] Master 127.24.60.254:45219 requested a full tablet report, sending...
I20260810 07:24:24.903441 24877 ts_manager.cc:194] Registered new tserver with Master: faae9474524c4a138f7e54e544637dc6 (127.24.60.193:44027)
I20260810 07:24:24.904443 24877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:48813
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:24.929004 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45219
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:25.015473 25088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:25.015722 25088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:25.015753 25088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:25.016970 25088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:25.017012 25088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:25.017045 25088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:25.018590 25088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:25.018800 25088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:25.018994 25088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:25.024422 25093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:25.024436 25094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:25.024436 25096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:25.024930 25088 server_base.cc:1034] running on GCE node
I20260810 07:24:25.025128 25088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:25.025415 25088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:25.026582 25088 hybrid_clock.cc:648] HybridClock initialized: now 1786346665026569 us; error 30 us; skew 500 ppm
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:25.032716 25088 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:25.033294 25088 webserver.cc:460] Webserver started at http://127.24.60.194:36389/ using document root <none> and password file <none>
I20260810 07:24:25.033586 25088 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:25.033689 25088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:25.033852 25088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:25.035058 25088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "23dc71c19f97497e92b852a926a9e398"
format_stamp: "Formatted at 2026-08-10 07:24:25 on dist-test-slave-vxj2"
server_key: "07ad9431c61f1d591519995022776ae9"
server_key_iv: "5bceca02890591b95ae85486f8ca888d"
server_key_version: "encryptionkey@0"
I20260810 07:24:25.035478 25088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "23dc71c19f97497e92b852a926a9e398"
format_stamp: "Formatted at 2026-08-10 07:24:25 on dist-test-slave-vxj2"
server_key: "07ad9431c61f1d591519995022776ae9"
server_key_iv: "5bceca02890591b95ae85486f8ca888d"
server_key_version: "encryptionkey@0"
I20260810 07:24:25.037068 25088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:25.038033 25104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:25.038261 25088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:25.038383 25088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "23dc71c19f97497e92b852a926a9e398"
format_stamp: "Formatted at 2026-08-10 07:24:25 on dist-test-slave-vxj2"
server_key: "07ad9431c61f1d591519995022776ae9"
server_key_iv: "5bceca02890591b95ae85486f8ca888d"
server_key_version: "encryptionkey@0"
I20260810 07:24:25.038497 25088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:25.056173 25088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:25.058624 25088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:25.058810 25088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:25.059110 25088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:25.059486 25088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:25.059546 25088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:25.059592 25088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:25.059618 25088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:25.066589 25088 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:45771
I20260810 07:24:25.066645 25238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:45771 every 8 connection(s)
I20260810 07:24:25.067027 25088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:25.075066 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 25088
I20260810 07:24:25.075183 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:25.075377 24819 external_mini_cluster.cc:1428] Setting key 2d87be1bec3537733f33b37a085d40c3
I20260810 07:24:25.077666 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:25.067967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:32861 (local address 127.24.60.254:45219)
0810 07:24:25.068053 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:25.068056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:25.068227 (+ 171us) server_negotiation.cc:407] Connection header received
0810 07:24:25.068798 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:25.068800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:25.068813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:25.068876 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:25.070026 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.070544 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.071606 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.071822 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.071923 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:25.074367 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:25.074369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:25.074369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:25.074378 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:25.076087 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.076556 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.076557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.076558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.076627 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.076911 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.076912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.076912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.077039 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:25.077118 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:25.077469 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 07:24:25.077573 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:24:25.077924 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:25.067860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45219 (local address 127.24.60.194:32861)
0810 07:24:25.068082 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:25.068139 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:25.068687 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:25.068952 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:25.068956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:25.069664 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:25.069891 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:25.069898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.070657 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.070659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:25.071507 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:25.071509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.071936 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.071937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:25.072070 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:25.072653 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:25.072662 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:25.074238 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:25.076201 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:25.076211 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:25.076214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:25.076465 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:25.076700 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:25.076701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:25.076702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:25.076824 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:25.077108 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:25.077116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:25.077320 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 07:24:25.077445 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 07:24:25.077626 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":112,"threads_started":1}
I20260810 07:24:25.078434 25241 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45219
I20260810 07:24:25.078570 25241 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:25.078819 25241 heartbeater.cc:511] Master 127.24.60.254:45219 requested a full tablet report, sending...
I20260810 07:24:25.079282 24877 ts_manager.cc:194] Registered new tserver with Master: 23dc71c19f97497e92b852a926a9e398 (127.24.60.194:45771)
I20260810 07:24:25.079667 24877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:32861
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:25.104730 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45219
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:25.202226 25247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:25.202502 25247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:25.202550 25247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:25.203864 25247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:25.203941 25247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:25.203991 25247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:25.205792 25247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:25.206106 25247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:25.206370 25247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:25.212345 25256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:25.212359 25253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:25.212373 25254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:25.212652 25247 server_base.cc:1034] running on GCE node
I20260810 07:24:25.212944 25247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:25.213253 25247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:25.214429 25247 hybrid_clock.cc:648] HybridClock initialized: now 1786346665214397 us; error 36 us; skew 500 ppm
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:25.217257 25247 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:25.217703 25247 webserver.cc:460] Webserver started at http://127.24.60.195:44297/ using document root <none> and password file <none>
I20260810 07:24:25.217927 25247 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:25.217972 25247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:25.218066 25247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:25.219094 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "bce1346297054dd8ad2d11b9efbecd1a"
format_stamp: "Formatted at 2026-08-10 07:24:25 on dist-test-slave-vxj2"
server_key: "0fb32558a3f660eff39b4f78b6ff2973"
server_key_iv: "894cfb2ebdb6ffe573cc27925e0e2c33"
server_key_version: "encryptionkey@0"
I20260810 07:24:25.219394 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "bce1346297054dd8ad2d11b9efbecd1a"
format_stamp: "Formatted at 2026-08-10 07:24:25 on dist-test-slave-vxj2"
server_key: "0fb32558a3f660eff39b4f78b6ff2973"
server_key_iv: "894cfb2ebdb6ffe573cc27925e0e2c33"
server_key_version: "encryptionkey@0"
I20260810 07:24:25.220849 25247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:25.221961 25265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:25.222218 25247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:25.222343 25247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "bce1346297054dd8ad2d11b9efbecd1a"
format_stamp: "Formatted at 2026-08-10 07:24:25 on dist-test-slave-vxj2"
server_key: "0fb32558a3f660eff39b4f78b6ff2973"
server_key_iv: "894cfb2ebdb6ffe573cc27925e0e2c33"
server_key_version: "encryptionkey@0"
I20260810 07:24:25.222441 25247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:25.246941 25247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:25.249682 25247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:25.249881 25247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:25.250165 25247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:25.250515 25247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:25.250571 25247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:25.250609 25247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:25.250658 25247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:25.257350 25247 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:39139
I20260810 07:24:25.257414 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:39139 every 8 connection(s)
I20260810 07:24:25.257773 25247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:25.262441 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 25247
I20260810 07:24:25.262555 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:25.262768 24819 external_mini_cluster.cc:1428] Setting key 25990f7289dc4ac5d9b165529cd50359
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:25.269021 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:25.258671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:33435 (local address 127.24.60.254:45219)
0810 07:24:25.258757 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:25.258760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:25.259187 (+ 427us) server_negotiation.cc:407] Connection header received
0810 07:24:25.259714 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:25.259715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:25.259727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:25.259794 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:25.260817 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.261519 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.262687 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.263027 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.263119 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:25.266138 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:25.266140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:25.266141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:25.266148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:25.267570 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.268037 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.268038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.268039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.268097 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.268383 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.268384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.268384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.268515 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:25.268578 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:25.268871 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 07:24:25.268950 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:24:25.269297 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:25.258639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45219 (local address 127.24.60.195:33435)
0810 07:24:25.258998 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:25.259057 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:25.259613 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:25.259877 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:25.259881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:25.260509 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:25.260719 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:25.260725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.261610 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.261611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:25.262528 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:25.262531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.263262 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.263263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:25.263498 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:25.264194 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:25.264204 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:25.266034 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:25.267657 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:25.267665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:25.267667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:25.267964 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:25.268201 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:25.268202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:25.268202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:25.268317 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:25.268584 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:25.268589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:25.268723 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 07:24:25.268816 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:24:25.269006 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":127,"threads_started":1}
I20260810 07:24:25.269870 25392 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45219
I20260810 07:24:25.270002 25392 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:25.270224 25392 heartbeater.cc:511] Master 127.24.60.254:45219 requested a full tablet report, sending...
I20260810 07:24:25.270644 24877 ts_manager.cc:194] Registered new tserver with Master: bce1346297054dd8ad2d11b9efbecd1a (127.24.60.195:39139)
I20260810 07:24:25.271080 24877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:33435
I20260810 07:24:25.275507 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:25.303499 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:25.294067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.24.60.193:44027)
0810 07:24:25.294280 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:25.294290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:25.294304 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:24:25.294335 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:25.294337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:25.294370 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:25.294464 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:25.295230 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.296086 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.297069 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.297512 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.297587 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:25.300447 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:25.300459 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:25.300459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:25.300467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:25.302222 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.302592 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.302596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.302596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.302674 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.302931 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.302932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.302932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.303112 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:25.303253 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:25.303320 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 07:24:25.303405 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":125,"threads_started":1}
W20260810 07:24:25.303972 24993 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39226
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[24827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:25.329344 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:25.321724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.24.60.193:44027)
0810 07:24:25.321792 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:25.321797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:25.321814 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:24:25.321843 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:25.321844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:25.321857 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:25.321930 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:25.322533 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.323076 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.323850 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:25.324095 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:25.324167 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:25.326913 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:25.326915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:25.326915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:25.326926 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:25.328274 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.328640 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.328641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.328641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.328692 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:25.328866 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:25.328867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:25.328867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:25.328966 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:25.329062 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:25.329141 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 07:24:25.329267 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:24:25.330014 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 24943
I20260810 07:24:25.339358 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 25088
I20260810 07:24:25.349870 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 25247
I20260810 07:24:25.357748 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 24843
2026-08-10T07:24:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2099 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[25407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[25407](info): set up 2 sockets
Aug 10 07:24:25 dist-test-slave-vxj2 krb5kdc[25407](info): commencing operation
krb5kdc: starting...
W20260810 07:24:26.499207 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:26 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:26.636968 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:41051
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:41051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:26.722131 25423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:26.722437 25423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:26.722481 25423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:26.723730 25423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:26.723809 25423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:26.723831 25423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:26.723852 25423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:26.723882 25423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:26.725502 25423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:41051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:41051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:26.725813 25423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:26.726064 25423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:26.731336 25432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:26.731310 25430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:26.731554 25423 server_base.cc:1034] running on GCE node
W20260810 07:24:26.731312 25429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:26.731981 25423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:26.732379 25423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:26.733577 25423 hybrid_clock.cc:648] HybridClock initialized: now 1786346666733561 us; error 30 us; skew 500 ppm
Aug 10 07:24:26 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346666, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:26.736407 25423 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:26.736976 25423 webserver.cc:460] Webserver started at http://127.24.60.254:43363/ using document root <none> and password file <none>
I20260810 07:24:26.737308 25423 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:26.737386 25423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:26.737553 25423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:26.738691 25423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "f36e0e350e54436190515ef66d0d68d7"
format_stamp: "Formatted at 2026-08-10 07:24:26 on dist-test-slave-vxj2"
server_key: "95ae49dd5953eb778803e04943467173"
server_key_iv: "3cea7b05982b2a647cafb6a611086d6a"
server_key_version: "encryptionkey@0"
I20260810 07:24:26.739145 25423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "f36e0e350e54436190515ef66d0d68d7"
format_stamp: "Formatted at 2026-08-10 07:24:26 on dist-test-slave-vxj2"
server_key: "95ae49dd5953eb778803e04943467173"
server_key_iv: "3cea7b05982b2a647cafb6a611086d6a"
server_key_version: "encryptionkey@0"
I20260810 07:24:26.740664 25423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:26.741645 25441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:26.741961 25423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:24:26.742132 25423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "f36e0e350e54436190515ef66d0d68d7"
format_stamp: "Formatted at 2026-08-10 07:24:26 on dist-test-slave-vxj2"
server_key: "95ae49dd5953eb778803e04943467173"
server_key_iv: "3cea7b05982b2a647cafb6a611086d6a"
server_key_version: "encryptionkey@0"
I20260810 07:24:26.742219 25423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:26.754195 25423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:26.756713 25423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:26.756964 25423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:26.761375 25423 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:41051
I20260810 07:24:26.761444 25500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:41051 every 8 connection(s)
I20260810 07:24:26.761866 25423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:26.762796 25501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:26.763000 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 25423
I20260810 07:24:26.763105 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:26.763309 24819 external_mini_cluster.cc:1428] Setting key bf8463f77379c15da229ca63696c5b59
I20260810 07:24:26.766425 25501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7: Bootstrap starting.
I20260810 07:24:26.767202 25501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:26.767520 25501 log.cc:824] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:26.768356 25501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7: No bootstrap required, opened a new log
Aug 10 07:24:26 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:26.771255 25501 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36e0e350e54436190515ef66d0d68d7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41051 } }
I20260810 07:24:26.771438 25501 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:26.771484 25501 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36e0e350e54436190515ef66d0d68d7, State: Initialized, Role: FOLLOWER
I20260810 07:24:26.771641 25501 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36e0e350e54436190515ef66d0d68d7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41051 } }
I20260810 07:24:26.771739 25501 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:26.771785 25501 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:26.771840 25501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:26.772643 25501 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36e0e350e54436190515ef66d0d68d7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41051 } }
I20260810 07:24:26.772764 25501 leader_election.cc:302] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36e0e350e54436190515ef66d0d68d7; no voters:
I20260810 07:24:26.772984 25501 leader_election.cc:288] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:26.773017 25507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:26.773126 25507 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [term 1 LEADER]: Becoming Leader. State: Replica: f36e0e350e54436190515ef66d0d68d7, State: Running, Role: LEADER
I20260810 07:24:26.773241 25507 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36e0e350e54436190515ef66d0d68d7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41051 } }
I20260810 07:24:26.773303 25501 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:26.773571 25507 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36e0e350e54436190515ef66d0d68d7. Latest consensus state: current_term: 1 leader_uuid: "f36e0e350e54436190515ef66d0d68d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36e0e350e54436190515ef66d0d68d7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41051 } } }
I20260810 07:24:26.773643 25507 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:26.773589 25508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36e0e350e54436190515ef66d0d68d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36e0e350e54436190515ef66d0d68d7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41051 } } }
I20260810 07:24:26.773862 25513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:26.773939 25508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:26.774470 25513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:26.775866 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:26.763924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.24.60.254:41051)
0810 07:24:26.764127 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:26.764137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:26.764170 (+ 33us) server_negotiation.cc:407] Connection header received
0810 07:24:26.764655 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:26.764664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:26.764726 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:26.764856 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:26.765726 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:26.766721 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:26.767864 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:26.768385 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:26.768533 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:26.771551 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:26.771556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:26.771556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:26.771570 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:26.773950 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:26.774445 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:26.774452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:26.774452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:26.774564 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:26.774871 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:26.774873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:26.774873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:26.775170 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:26.775380 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:26.775474 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 07:24:26.775622 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260810 07:24:26.776355 25513 catalog_manager.cc:1347] Generated new cluster ID: 4dc2a9ebfe0b40799ae88ef98a16014a
I20260810 07:24:26.776413 25513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:26.788244 25513 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:26.788914 25513 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:26.794121 25513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36e0e350e54436190515ef66d0d68d7: Generated new TSK 0
I20260810 07:24:26.794355 25513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:26.831954 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41051
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:26.917716 25531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:26.917948 25531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:26.917976 25531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:26.919212 25531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:26.919255 25531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:26.919283 25531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:26.920975 25531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41051
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:26.921288 25531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:26.921568 25531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:26.927425 25537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:26.927417 25539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:26.927417 25536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:26.927712 25531 server_base.cc:1034] running on GCE node
I20260810 07:24:26.927976 25531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:26.928210 25531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:26.929354 25531 hybrid_clock.cc:648] HybridClock initialized: now 1786346666929340 us; error 29 us; skew 500 ppm
Aug 10 07:24:26 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346666, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:26.932199 25531 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:26.932724 25531 webserver.cc:460] Webserver started at http://127.24.60.193:33027/ using document root <none> and password file <none>
I20260810 07:24:26.932996 25531 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:26.933077 25531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:26.933264 25531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:26.934422 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "17845282a0df4e18a3adde2d9dd224e2"
format_stamp: "Formatted at 2026-08-10 07:24:26 on dist-test-slave-vxj2"
server_key: "469423f872dcdc1d4956f749d8f20da7"
server_key_iv: "381dc432e4ff92eafe28582b5e604001"
server_key_version: "encryptionkey@0"
I20260810 07:24:26.934819 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "17845282a0df4e18a3adde2d9dd224e2"
format_stamp: "Formatted at 2026-08-10 07:24:26 on dist-test-slave-vxj2"
server_key: "469423f872dcdc1d4956f749d8f20da7"
server_key_iv: "381dc432e4ff92eafe28582b5e604001"
server_key_version: "encryptionkey@0"
I20260810 07:24:26.936362 25531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:26.937316 25547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:26.937516 25531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:26.937623 25531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "17845282a0df4e18a3adde2d9dd224e2"
format_stamp: "Formatted at 2026-08-10 07:24:26 on dist-test-slave-vxj2"
server_key: "469423f872dcdc1d4956f749d8f20da7"
server_key_iv: "381dc432e4ff92eafe28582b5e604001"
server_key_version: "encryptionkey@0"
I20260810 07:24:26.937716 25531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:26.946233 25531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:26.948333 25531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:26.948520 25531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:26.948766 25531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:26.949108 25531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:26.949196 25531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:26.949245 25531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:26.949271 25531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:26.955515 25531 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46363
I20260810 07:24:26.955557 25677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46363 every 8 connection(s)
I20260810 07:24:26.955968 25531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:26.957510 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 25531
I20260810 07:24:26.957612 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:26.957784 24819 external_mini_cluster.cc:1428] Setting key 6cbe09d258f6f637637cdd63f2d8278d
Aug 10 07:24:26 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346666, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
I20260810 07:24:26.967375 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:26.956810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:55285 (local address 127.24.60.254:41051)
0810 07:24:26.956882 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:26.956884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:26.957113 (+ 229us) server_negotiation.cc:407] Connection header received
0810 07:24:26.958583 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:26.958584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:26.958594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:26.958638 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:26.959617 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:26.960389 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:26.961526 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:26.961754 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:26.961850 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:26.964069 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:26.964070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:26.964071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:26.964079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:26.965756 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:26.966232 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:26.966233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:26.966233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:26.966298 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:26.966647 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:26.966648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:26.966648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:26.966777 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:26.966853 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:26.967177 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 07:24:26.967275 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:24:26.967622 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:26.956785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41051 (local address 127.24.60.193:55285)
0810 07:24:26.956979 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:26.957017 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:26.958439 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:26.958725 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:26.958731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:26.959141 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:26.959425 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:26.959432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:26.960506 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:26.960507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:26.961426 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:26.961429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:26.961864 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:26.961865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:26.962013 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:26.962583 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:26.962591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:26.963966 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:26.965876 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:26.965885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:26.965888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:26.966131 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:26.966396 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:26.966397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:26.966398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:26.966573 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:26.966856 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:26.966865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:26.967035 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 07:24:26.967156 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 07:24:26.967329 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260810 07:24:26.968186 25678 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41051
I20260810 07:24:26.968325 25678 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:26.968529 25678 heartbeater.cc:511] Master 127.24.60.254:41051 requested a full tablet report, sending...
I20260810 07:24:26.969228 25460 ts_manager.cc:194] Registered new tserver with Master: 17845282a0df4e18a3adde2d9dd224e2 (127.24.60.193:46363)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
I20260810 07:24:26.970156 25460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:55285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:26.986193 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41051
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:27.074103 25683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:27.074342 25683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:27.074369 25683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:27.075647 25683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:27.075699 25683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:27.075728 25683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:27.077312 25683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41051
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:27.077524 25683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:27.077713 25683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:27.083150 25692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:27.083164 25689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:27.083164 25690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:27.083449 25683 server_base.cc:1034] running on GCE node
I20260810 07:24:27.083726 25683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:27.084059 25683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:27.085227 25683 hybrid_clock.cc:648] HybridClock initialized: now 1786346667085196 us; error 48 us; skew 500 ppm
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:27.088130 25683 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:27.088626 25683 webserver.cc:460] Webserver started at http://127.24.60.194:34693/ using document root <none> and password file <none>
I20260810 07:24:27.088864 25683 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:27.088912 25683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:27.089012 25683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:27.090220 25683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "2794c58e79ec4685bfc0cc5d35f83f33"
format_stamp: "Formatted at 2026-08-10 07:24:27 on dist-test-slave-vxj2"
server_key: "e666bf76ffaf8745c0b57d60f39c6af7"
server_key_iv: "7f371a415efb58c30e6ef2f72267afb4"
server_key_version: "encryptionkey@0"
I20260810 07:24:27.090559 25683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "2794c58e79ec4685bfc0cc5d35f83f33"
format_stamp: "Formatted at 2026-08-10 07:24:27 on dist-test-slave-vxj2"
server_key: "e666bf76ffaf8745c0b57d60f39c6af7"
server_key_iv: "7f371a415efb58c30e6ef2f72267afb4"
server_key_version: "encryptionkey@0"
I20260810 07:24:27.092087 25683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:27.092954 25699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:27.093179 25683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:27.093273 25683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "2794c58e79ec4685bfc0cc5d35f83f33"
format_stamp: "Formatted at 2026-08-10 07:24:27 on dist-test-slave-vxj2"
server_key: "e666bf76ffaf8745c0b57d60f39c6af7"
server_key_iv: "7f371a415efb58c30e6ef2f72267afb4"
server_key_version: "encryptionkey@0"
I20260810 07:24:27.093338 25683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:27.112089 25683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:27.114666 25683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:27.114794 25683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:27.114997 25683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:27.115357 25683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:27.115398 25683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:27.115429 25683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:27.115442 25683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:27.121668 25683 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:36391
I20260810 07:24:27.121773 25831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:36391 every 8 connection(s)
I20260810 07:24:27.122159 25683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:27.132016 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 25683
I20260810 07:24:27.132139 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:27.132243 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.123010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:41591 (local address 127.24.60.254:41051)
0810 07:24:27.123091 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.123093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.123387 (+ 294us) server_negotiation.cc:407] Connection header received
0810 07:24:27.124408 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.124409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.124422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.124478 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:27.125226 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.125747 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.126841 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.127057 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.127135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.129440 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:27.129442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:27.129442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:27.129450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:27.130732 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.131205 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.131206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.131207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.131273 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.131530 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.131531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.131532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.131661 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:27.131737 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.132057 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.132150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:24:27.132354 24819 external_mini_cluster.cc:1428] Setting key cc4c955cd585ad6fea9f574ad9b640dd
I20260810 07:24:27.132493 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.123038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41051 (local address 127.24.60.194:41591)
0810 07:24:27.123230 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.123270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.124281 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.124554 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.124563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.124837 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:27.125082 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.125086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.125864 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.125865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.126743 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.126745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.127145 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.127146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.127279 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.127832 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:27.127885 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:27.129335 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:27.130821 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:27.130829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:27.130832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:27.131121 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:27.131343 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:27.131344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:27.131344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:27.131461 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:27.131743 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:27.131751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:27.131903 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.132021 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.132198 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260810 07:24:27.133013 25832 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41051
I20260810 07:24:27.133128 25832 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:27.133373 25832 heartbeater.cc:511] Master 127.24.60.254:41051 requested a full tablet report, sending...
I20260810 07:24:27.133819 25460 ts_manager.cc:194] Registered new tserver with Master: 2794c58e79ec4685bfc0cc5d35f83f33 (127.24.60.194:36391)
I20260810 07:24:27.134205 25460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:41591
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:27.161000 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41051
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:27.250923 25837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:27.251206 25837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:27.251238 25837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:27.252656 25837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:27.252729 25837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:27.252771 25837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:27.254503 25837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41051
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:27.254755 25837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:27.254998 25837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:27.260906 25843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:27.261056 25845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:27.260914 25842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:27.261152 25837 server_base.cc:1034] running on GCE node
I20260810 07:24:27.261402 25837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:27.261713 25837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:27.262926 25837 hybrid_clock.cc:648] HybridClock initialized: now 1786346667262931 us; error 62 us; skew 500 ppm
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:27.265919 25837 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:27.266494 25837 webserver.cc:460] Webserver started at http://127.24.60.195:45315/ using document root <none> and password file <none>
I20260810 07:24:27.266775 25837 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:27.266848 25837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:27.267011 25837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:27.268236 25837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "a3970af87964462e93b4f89841c41dc5"
format_stamp: "Formatted at 2026-08-10 07:24:27 on dist-test-slave-vxj2"
server_key: "26caea0f04ca0f20105d255bfce2e61c"
server_key_iv: "efcf2cf1304d88e006232858f32f28ea"
server_key_version: "encryptionkey@0"
I20260810 07:24:27.268644 25837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "a3970af87964462e93b4f89841c41dc5"
format_stamp: "Formatted at 2026-08-10 07:24:27 on dist-test-slave-vxj2"
server_key: "26caea0f04ca0f20105d255bfce2e61c"
server_key_iv: "efcf2cf1304d88e006232858f32f28ea"
server_key_version: "encryptionkey@0"
I20260810 07:24:27.270251 25837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:27.271149 25854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:27.271353 25837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:27.271471 25837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "a3970af87964462e93b4f89841c41dc5"
format_stamp: "Formatted at 2026-08-10 07:24:27 on dist-test-slave-vxj2"
server_key: "26caea0f04ca0f20105d255bfce2e61c"
server_key_iv: "efcf2cf1304d88e006232858f32f28ea"
server_key_version: "encryptionkey@0"
I20260810 07:24:27.271557 25837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:27.288331 25837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:27.290596 25837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:27.290786 25837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:27.291064 25837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:27.291390 25837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:27.291445 25837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:27.291488 25837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:27.291513 25837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:27.298535 25837 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:38453
I20260810 07:24:27.298584 25980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:38453 every 8 connection(s)
I20260810 07:24:27.298933 25837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:27.306881 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 25837
I20260810 07:24:27.306972 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:27.307144 24819 external_mini_cluster.cc:1428] Setting key 0ce0c0252ee0250a3a770f71d6c8cc36
I20260810 07:24:27.308948 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.299755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:54015 (local address 127.24.60.254:41051)
0810 07:24:27.299825 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.299827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.300045 (+ 218us) server_negotiation.cc:407] Connection header received
0810 07:24:27.300982 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.300983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.300994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.301059 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:27.301793 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.302300 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.303303 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.303514 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.303570 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.305824 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:27.305826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:27.305826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:27.305835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:27.307479 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.307915 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.307916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.307916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.307982 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.308256 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.308257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.308257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.308383 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:27.308466 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.308758 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.308851 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:24:27.309206 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.299734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41051 (local address 127.24.60.195:54015)
0810 07:24:27.299896 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.299942 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.300883 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.301121 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.301125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.301483 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:27.301690 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.301696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.302392 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.302393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.303221 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.303223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.303594 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.303595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.303700 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.304193 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:27.304202 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:27.305710 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:27.307563 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:27.307574 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:27.307576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:27.307831 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:27.308055 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:27.308055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:27.308056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:27.308180 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:27.308455 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:27.308463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:27.308617 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.308737 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.308927 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260810 07:24:27.309731 25982 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41051
I20260810 07:24:27.309872 25982 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:27.310143 25982 heartbeater.cc:511] Master 127.24.60.254:41051 requested a full tablet report, sending...
I20260810 07:24:27.310582 25460 ts_manager.cc:194] Registered new tserver with Master: a3970af87964462e93b4f89841c41dc5 (127.24.60.195:38453)
I20260810 07:24:27.310972 25460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:54015
I20260810 07:24:27.319309 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:24:27.327874 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.321280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.24.60.254:41051)
0810 07:24:27.321362 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.321364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.321454 (+ 90us) server_negotiation.cc:407] Connection header received
0810 07:24:27.321506 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.321506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.321517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.321578 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:27.322176 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.322743 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.323521 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.323777 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.323840 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.325093 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:27.325094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:27.325094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:27.325100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:27.326706 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.327089 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.327090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.327090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.327154 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.327326 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.327326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.327327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.327484 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:27.327590 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.327658 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.327746 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:24:27.330350 25460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:27.331984 25460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:27.339123 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.334606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46363 (local address 127.0.0.1:60584)
0810 07:24:27.334803 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.334848 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.334970 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.335359 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.335360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.335363 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:27.335437 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.335439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.336504 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.336505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.337716 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.337719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.338733 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.338734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.338834 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.338850 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.338956 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.339019 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260810 07:24:27.339123 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.334726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.24.60.193:46363)
0810 07:24:27.335126 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.335130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.335139 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:27.335177 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.335178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.335197 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.335271 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:27.335576 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.336410 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.337790 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.338639 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.338722 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.338786 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.338975 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.339021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":116,"threads_started":1}
I20260810 07:24:27.339540 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.334608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:38453 (local address 127.0.0.1:38944)
0810 07:24:27.334976 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.334991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.335051 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.335296 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.335300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.335302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:27.335392 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.335395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.336959 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.336960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.338015 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.338017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.339256 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.339257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.339348 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.339353 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.339410 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.339468 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260810 07:24:27.339744 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.335128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.24.60.194:36391)
0810 07:24:27.335627 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.335632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.335643 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:27.335684 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.335686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.335703 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.335792 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:27.335925 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.337079 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.338615 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.339164 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.339257 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.339321 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.339579 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.339650 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":117,"threads_started":1}
I20260810 07:24:27.339816 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.334789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.24.60.195:38453)
0810 07:24:27.335018 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.335024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.335037 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:24:27.335148 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.335151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.335169 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.335257 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:27.335508 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.336829 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.338102 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.338644 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.338721 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.338779 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.339669 (+ 890us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.339724 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
I20260810 07:24:27.340000 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.334856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:36391 (local address 127.0.0.1:57760)
0810 07:24:27.335416 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.335427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.335479 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.335803 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.335803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.335806 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:27.335852 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.335854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.337228 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.337230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.338513 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.338516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.339364 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.339365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.339423 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.339428 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.339480 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.339521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":76,"threads_started":1}
I20260810 07:24:27.340106 25601 tablet_service.cc:1467] Processing CreateTablet for tablet f2011eae28a84d3287dd3ca90b91804f (DEFAULT_TABLE table=test-table [id=b9307fdd8b7a44d39de42bcc4c89ae04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:27.340554 25601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2011eae28a84d3287dd3ca90b91804f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:27.343780 26002 tablet_bootstrap.cc:492] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2: Bootstrap starting.
I20260810 07:24:27.344625 25910 tablet_service.cc:1467] Processing CreateTablet for tablet f2011eae28a84d3287dd3ca90b91804f (DEFAULT_TABLE table=test-table [id=b9307fdd8b7a44d39de42bcc4c89ae04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:27.344765 26002 tablet_bootstrap.cc:654] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:27.344856 25757 tablet_service.cc:1467] Processing CreateTablet for tablet f2011eae28a84d3287dd3ca90b91804f (DEFAULT_TABLE table=test-table [id=b9307fdd8b7a44d39de42bcc4c89ae04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:27.345034 25910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2011eae28a84d3287dd3ca90b91804f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:27.345269 25757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2011eae28a84d3287dd3ca90b91804f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:27.345475 26002 log.cc:824] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:27.346357 26002 tablet_bootstrap.cc:492] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2: No bootstrap required, opened a new log
I20260810 07:24:27.346429 26002 ts_tablet_manager.cc:1397] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:24:27.348583 26004 tablet_bootstrap.cc:492] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5: Bootstrap starting.
I20260810 07:24:27.348692 26002 raft_consensus.cc:348] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.348901 26002 raft_consensus.cc:374] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:27.348943 26002 raft_consensus.cc:729] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17845282a0df4e18a3adde2d9dd224e2, State: Initialized, Role: FOLLOWER
I20260810 07:24:27.349088 26002 consensus_queue.cc:260] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.349468 26002 ts_tablet_manager.cc:1428] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:27.349614 25678 heartbeater.cc:503] Master 127.24.60.254:41051 was elected leader, sending a full tablet report...
I20260810 07:24:27.349784 26004 tablet_bootstrap.cc:654] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:27.350098 26005 tablet_bootstrap.cc:492] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: Bootstrap starting.
I20260810 07:24:27.350775 26005 tablet_bootstrap.cc:654] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:27.351186 26004 log.cc:824] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:27.351410 26005 log.cc:824] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:27.352219 26005 tablet_bootstrap.cc:492] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: No bootstrap required, opened a new log
I20260810 07:24:27.352236 26004 tablet_bootstrap.cc:492] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5: No bootstrap required, opened a new log
I20260810 07:24:27.352308 26004 ts_tablet_manager.cc:1397] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:24:27.352309 26005 ts_tablet_manager.cc:1397] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:24:27.353921 26005 raft_consensus.cc:348] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.354113 26005 raft_consensus.cc:374] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:27.354017 26004 raft_consensus.cc:348] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.354162 26005 raft_consensus.cc:729] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794c58e79ec4685bfc0cc5d35f83f33, State: Initialized, Role: FOLLOWER
I20260810 07:24:27.354210 26004 raft_consensus.cc:374] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:27.354261 26004 raft_consensus.cc:729] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3970af87964462e93b4f89841c41dc5, State: Initialized, Role: FOLLOWER
I20260810 07:24:27.354318 26005 consensus_queue.cc:260] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.354405 26004 consensus_queue.cc:260] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.354696 25832 heartbeater.cc:503] Master 127.24.60.254:41051 was elected leader, sending a full tablet report...
I20260810 07:24:27.354772 25982 heartbeater.cc:503] Master 127.24.60.254:41051 was elected leader, sending a full tablet report...
I20260810 07:24:27.354792 26004 ts_tablet_manager.cc:1428] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 07:24:27.354960 26005 ts_tablet_manager.cc:1428] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:27.362501 26007 raft_consensus.cc:482] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:27.362669 26007 raft_consensus.cc:504] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.363076 26007 leader_election.cc:288] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2794c58e79ec4685bfc0cc5d35f83f33 (127.24.60.194:36391), a3970af87964462e93b4f89841c41dc5 (127.24.60.195:38453)
I20260810 07:24:27.366276 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.363361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:44035 (local address 127.24.60.194:36391)
0810 07:24:27.363423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.363425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.363628 (+ 203us) server_negotiation.cc:407] Connection header received
0810 07:24:27.363661 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.363663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.363673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.363743 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:27.364016 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.364527 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.365553 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.365916 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.366007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.366016 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.366157 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.366202 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:24:27.366282 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.363285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:38453 (local address 127.24.60.193:53039)
0810 07:24:27.363352 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.363366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.363410 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.363644 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.363645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.363648 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:27.363740 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.363743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.364474 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.364476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.365438 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.365440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.366002 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.366003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.366090 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.366093 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.366163 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.366206 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260810 07:24:27.366299 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.363379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53039 (local address 127.24.60.195:38453)
0810 07:24:27.363423 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.363425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.363435 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:27.363469 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.363470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.363480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.363570 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:27.363823 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.364384 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.365529 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.365915 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.366001 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.366011 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.366182 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.366226 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:24:27.366299 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.363285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:36391 (local address 127.24.60.193:44035)
0810 07:24:27.363507 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:27.363518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:27.363587 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:27.363762 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:27.363763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:27.363764 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:27.363920 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.363922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.364611 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.364611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.365446 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:27.365448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.366005 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.366006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:27.366083 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.366087 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:27.366131 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 07:24:27.366193 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":26,"thread_start_us":124,"threads_started":1}
I20260810 07:24:27.366698 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2011eae28a84d3287dd3ca90b91804f" candidate_uuid: "17845282a0df4e18a3adde2d9dd224e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3970af87964462e93b4f89841c41dc5" is_pre_election: true
I20260810 07:24:27.366891 25931 raft_consensus.cc:2454] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17845282a0df4e18a3adde2d9dd224e2 in term 0.
I20260810 07:24:27.366930 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2011eae28a84d3287dd3ca90b91804f" candidate_uuid: "17845282a0df4e18a3adde2d9dd224e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" is_pre_election: true
I20260810 07:24:27.367084 25780 raft_consensus.cc:2454] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17845282a0df4e18a3adde2d9dd224e2 in term 0.
I20260810 07:24:27.367158 25548 leader_election.cc:302] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17845282a0df4e18a3adde2d9dd224e2, a3970af87964462e93b4f89841c41dc5; no voters:
I20260810 07:24:27.367297 26007 raft_consensus.cc:2781] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:27.367350 26007 raft_consensus.cc:482] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:27.367385 26007 raft_consensus.cc:3037] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:27.368096 26007 raft_consensus.cc:504] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.368276 26007 leader_election.cc:288] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [CANDIDATE]: Term 1 election: Requested vote from peers 2794c58e79ec4685bfc0cc5d35f83f33 (127.24.60.194:36391), a3970af87964462e93b4f89841c41dc5 (127.24.60.195:38453)
I20260810 07:24:27.368467 25931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2011eae28a84d3287dd3ca90b91804f" candidate_uuid: "17845282a0df4e18a3adde2d9dd224e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3970af87964462e93b4f89841c41dc5"
I20260810 07:24:27.368470 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2011eae28a84d3287dd3ca90b91804f" candidate_uuid: "17845282a0df4e18a3adde2d9dd224e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794c58e79ec4685bfc0cc5d35f83f33"
I20260810 07:24:27.368559 25931 raft_consensus.cc:3037] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:27.368556 25780 raft_consensus.cc:3037] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:27.369405 25931 raft_consensus.cc:2454] T f2011eae28a84d3287dd3ca90b91804f P a3970af87964462e93b4f89841c41dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17845282a0df4e18a3adde2d9dd224e2 in term 1.
I20260810 07:24:27.369563 25780 raft_consensus.cc:2454] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17845282a0df4e18a3adde2d9dd224e2 in term 1.
I20260810 07:24:27.369658 25548 leader_election.cc:302] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17845282a0df4e18a3adde2d9dd224e2, a3970af87964462e93b4f89841c41dc5; no voters:
I20260810 07:24:27.369774 26007 raft_consensus.cc:2781] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:27.369951 26007 raft_consensus.cc:686] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [term 1 LEADER]: Becoming Leader. State: Replica: 17845282a0df4e18a3adde2d9dd224e2, State: Running, Role: LEADER
I20260810 07:24:27.370033 26007 consensus_queue.cc:237] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } }
I20260810 07:24:27.371078 25458 catalog_manager.cc:5495] T f2011eae28a84d3287dd3ca90b91804f P 17845282a0df4e18a3adde2d9dd224e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17845282a0df4e18a3adde2d9dd224e2 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "17845282a0df4e18a3adde2d9dd224e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17845282a0df4e18a3adde2d9dd224e2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2794c58e79ec4685bfc0cc5d35f83f33" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3970af87964462e93b4f89841c41dc5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38453 } health_report { overall_health: UNKNOWN } } }
W20260810 07:24:27.372743 25833 tablet.cc:2369] T f2011eae28a84d3287dd3ca90b91804f P 2794c58e79ec4685bfc0cc5d35f83f33: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:27.386376 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.376164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.24.60.193:46363)
0810 07:24:27.376250 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.376252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.376314 (+ 62us) server_negotiation.cc:407] Connection header received
0810 07:24:27.376360 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.376360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.376369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.376422 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:27.377062 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.377613 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.378482 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.378673 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.378736 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.381846 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:27.381869 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:27.381869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:27.381880 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:27.384608 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.385211 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.385217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.385217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.385351 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.385705 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.385707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.385707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.385905 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:27.386095 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.386166 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.386274 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:27.414327 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.406187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.24.60.193:46363)
0810 07:24:27.406277 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.406279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.406366 (+ 87us) server_negotiation.cc:407] Connection header received
0810 07:24:27.406437 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.406437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.406447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.406519 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:27.407178 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.407699 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.408523 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.408737 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.408798 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.411448 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:27.411449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:27.411449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:27.411457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:27.413003 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.413461 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.413462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.413463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.413520 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.413724 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.413725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.413725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.413831 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:27.413908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.414183 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.414257 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[25407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:27.439760 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:27.431614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.24.60.193:46363)
0810 07:24:27.431685 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:27.431687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:27.431783 (+ 96us) server_negotiation.cc:407] Connection header received
0810 07:24:27.431828 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:27.431829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:27.431838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:27.431915 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:27.432564 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.433109 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.434006 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:27.434215 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:27.434276 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:27.436831 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:27.436833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:27.436834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:27.436847 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:27.438419 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.438880 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.438881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.438881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.438939 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:27.439163 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:27.439164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:27.439164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:27.439304 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:27.439411 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:27.439563 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 07:24:27.439652 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 07:24:27.440338 25577 tablet_service.cc:2908] Rejecting scan request for tablet f2011eae28a84d3287dd3ca90b91804f: Uninitialized: clean time has not yet been initialized
I20260810 07:24:27.441361 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 25531
I20260810 07:24:27.449545 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 25683
I20260810 07:24:27.456903 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 25837
I20260810 07:24:27.465243 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 25423
2026-08-10T07:24:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2107 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[26029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[26029](info): set up 2 sockets
Aug 10 07:24:27 dist-test-slave-vxj2 krb5kdc[26029](info): commencing operation
krb5kdc: starting...
W20260810 07:24:28.607657 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:28 dist-test-slave-vxj2 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:28.736680 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42577
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:42577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:28.821278 26044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:28.821588 26044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:28.821632 26044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:28.822875 26044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:28.822958 26044 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 07:24:28.822980 26044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:28.823002 26044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:28.823035 26044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:28.823057 26044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:28.824771 26044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:42577
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:28.825076 26044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:28.825363 26044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:28.830549 26049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:28.830621 26044 server_base.cc:1034] running on GCE node
W20260810 07:24:28.830575 26050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:28.830569 26052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:28.831099 26044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:28.831434 26044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:28.832592 26044 hybrid_clock.cc:648] HybridClock initialized: now 1786346668832578 us; error 33 us; skew 500 ppm
Aug 10 07:24:28 dist-test-slave-vxj2 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346668, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:28.835232 26044 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:28.835721 26044 webserver.cc:460] Webserver started at http://127.24.60.254:42071/ using document root <none> and password file <none>
I20260810 07:24:28.835961 26044 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:28.836007 26044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:28.836175 26044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:28.837378 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "05f3fe86ce3d4ed6b634a2cec87e2278"
format_stamp: "Formatted at 2026-08-10 07:24:28 on dist-test-slave-vxj2"
server_key: "d9d1cd0084a9973ee8073831f0e06cdb"
server_key_iv: "5c38c63cd0d3d5da372aae06a33f7a8e"
server_key_version: "encryptionkey@0"
I20260810 07:24:28.837760 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "05f3fe86ce3d4ed6b634a2cec87e2278"
format_stamp: "Formatted at 2026-08-10 07:24:28 on dist-test-slave-vxj2"
server_key: "d9d1cd0084a9973ee8073831f0e06cdb"
server_key_iv: "5c38c63cd0d3d5da372aae06a33f7a8e"
server_key_version: "encryptionkey@0"
I20260810 07:24:28.839221 26044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:28.840121 26059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:28.840349 26044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:28.840453 26044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "05f3fe86ce3d4ed6b634a2cec87e2278"
format_stamp: "Formatted at 2026-08-10 07:24:28 on dist-test-slave-vxj2"
server_key: "d9d1cd0084a9973ee8073831f0e06cdb"
server_key_iv: "5c38c63cd0d3d5da372aae06a33f7a8e"
server_key_version: "encryptionkey@0"
I20260810 07:24:28.840550 26044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:28.849269 26044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:28.851807 26044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:28.852008 26044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:28.857275 26044 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:42577
I20260810 07:24:28.857337 26128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:42577 every 8 connection(s)
I20260810 07:24:28.857746 26044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:28.858485 26130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:28.860898 26130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: Bootstrap starting.
I20260810 07:24:28.861548 26130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:28.861856 26130 log.cc:824] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:28.862264 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 26044
I20260810 07:24:28.862386 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:28.862576 24819 external_mini_cluster.cc:1428] Setting key f3fbe72aae83bd14c22d121bdaca46f1
I20260810 07:24:28.862700 26130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: No bootstrap required, opened a new log
I20260810 07:24:28.865422 26130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42577 } }
I20260810 07:24:28.865576 26130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:28.865599 26130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f3fe86ce3d4ed6b634a2cec87e2278, State: Initialized, Role: FOLLOWER
I20260810 07:24:28.865743 26130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42577 } }
I20260810 07:24:28.865824 26130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:28.865868 26130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:28.865918 26130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:28.866575 26130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42577 } }
I20260810 07:24:28.866724 26130 leader_election.cc:302] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f3fe86ce3d4ed6b634a2cec87e2278; no voters:
I20260810 07:24:28.866942 26130 leader_election.cc:288] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:28.866971 26137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:28.867094 26137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [term 1 LEADER]: Becoming Leader. State: Replica: 05f3fe86ce3d4ed6b634a2cec87e2278, State: Running, Role: LEADER
I20260810 07:24:28.867183 26137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42577 } }
I20260810 07:24:28.867244 26130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:28.867528 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f3fe86ce3d4ed6b634a2cec87e2278. Latest consensus state: current_term: 1 leader_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42577 } } }
I20260810 07:24:28.867602 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:28.867548 26138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f3fe86ce3d4ed6b634a2cec87e2278" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42577 } } }
I20260810 07:24:28.867758 26138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278 [sys.catalog]: This master's current role is: LEADER
W20260810 07:24:28.869292 26155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:24:28.869412 26155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:24:28.869592 26142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:28.869791 26142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 07:24:28 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:28.871476 26142 catalog_manager.cc:1347] Generated new cluster ID: e1e700320e384d499e508ef6685eb8b9
I20260810 07:24:28.871536 26142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:28.875362 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:28.863177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.24.60.254:42577)
0810 07:24:28.863409 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:28.863417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:28.863450 (+ 33us) server_negotiation.cc:407] Connection header received
0810 07:24:28.863997 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:28.864005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:28.864058 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:28.864214 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:28.864883 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:28.866107 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:28.866961 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:28.867580 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:28.868034 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:28.870825 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:28.870830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:28.870830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:28.870844 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:28.873014 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:28.873505 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:28.873508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:28.873509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:28.873615 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:28.873843 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:28.873844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:28.873844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:28.874174 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:28.874798 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:28.874884 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 07:24:28.875078 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":119,"threads_started":1}
I20260810 07:24:28.889230 26142 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:28.889820 26142 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:28.894310 26142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: Generated new TSK 0
I20260810 07:24:28.894565 26142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:28.928256 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42577
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:29.037712 26161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:29.037940 26161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:29.037966 26161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:29.039218 26161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:29.039266 26161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:29.039287 26161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:29.039307 26161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:29.040797 26161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:42577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:29.041059 26161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:29.041339 26161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:29.047320 26169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:29.047341 26167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:29.047323 26166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:29.047503 26161 server_base.cc:1034] running on GCE node
I20260810 07:24:29.047832 26161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:29.048108 26161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:29.049273 26161 hybrid_clock.cc:648] HybridClock initialized: now 1786346669049246 us; error 59 us; skew 500 ppm
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:29.052086 26161 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:29.052580 26161 webserver.cc:460] Webserver started at http://127.24.60.193:32815/ using document root <none> and password file <none>
I20260810 07:24:29.052829 26161 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:29.052901 26161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:29.053063 26161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:29.054217 26161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "0430f3d5032b2d6126026fc0f9b45639"
server_key_iv: "03fee8ac83405d108a51c3228a442ae3"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.054601 26161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "0430f3d5032b2d6126026fc0f9b45639"
server_key_iv: "03fee8ac83405d108a51c3228a442ae3"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.056113 26161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:29.056981 26176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.057235 26161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:29.057343 26161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "0430f3d5032b2d6126026fc0f9b45639"
server_key_iv: "03fee8ac83405d108a51c3228a442ae3"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.057431 26161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:29.071194 26161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:29.073900 26161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:29.074092 26161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:29.074697 26161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:29.074766 26161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.074820 26161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:29.074851 26161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.082249 26161 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:39437
I20260810 07:24:29.082367 26306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:39437 every 8 connection(s)
I20260810 07:24:29.082695 26161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:29.085215 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 26161
I20260810 07:24:29.085348 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:29.085536 24819 external_mini_cluster.cc:1428] Setting key 2e1ad9ff2901074b0c2845ead39e7c13
I20260810 07:24:29.086726 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.075360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:55955 (local address 127.24.60.254:42577)
0810 07:24:29.075412 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.075414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.075703 (+ 289us) server_negotiation.cc:407] Connection header received
0810 07:24:29.076307 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.076309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.076325 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.076406 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.077499 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.078366 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.080128 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.080511 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.080600 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.083381 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.083382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.083383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.083390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.084798 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.085428 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.085430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.085430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.085520 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.085891 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.085892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.085892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.086057 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.086129 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.086563 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.086658 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 07:24:29.087129 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.075360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.24.60.193:55955)
0810 07:24:29.075571 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.075610 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.076147 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.076542 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.076549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.076929 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.077370 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.077378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.078491 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.078491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.080021 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.080024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.080626 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.080627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.080800 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.081508 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.081519 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.083281 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.084895 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.084902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.084904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.085337 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.085609 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.085610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.085610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.085767 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.086150 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.086158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.086380 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.086508 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.086740 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260810 07:24:29.087735 26307 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42577
I20260810 07:24:29.087854 26307 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:29.088089 26307 heartbeater.cc:511] Master 127.24.60.254:42577 requested a full tablet report, sending...
I20260810 07:24:29.088691 26078 ts_manager.cc:194] Registered new tserver with Master: 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437)
I20260810 07:24:29.089690 26078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:55955
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
I20260810 07:24:29.096374 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.089012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.24.60.193:51289)
0810 07:24:29.089217 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.089228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.089318 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.089537 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.089538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.089801 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.089873 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.089876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.090771 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.090773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.091685 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.091688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.092295 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.092296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.092384 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.093053 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.093057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.093473 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.095303 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.095304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.095305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.095491 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.095712 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.095713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.095713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.095767 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.096085 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.096086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.096146 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.096205 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.096298 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":107,"threads_started":1}
I20260810 07:24:29.096457 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.089068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:51289 (local address 127.24.60.254:42577)
0810 07:24:29.089115 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.089117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.089306 (+ 189us) server_negotiation.cc:407] Connection header received
0810 07:24:29.089376 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.089377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.089390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.089445 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.089946 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.090643 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.091804 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.092177 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.092266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.093566 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.093568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.093568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.093576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.095206 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.095575 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.095576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.095577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.095640 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.095832 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.095833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.095833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.096004 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.096086 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.096259 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.096350 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:29.117439 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42577
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:29.211253 26319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:29.211566 26319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:29.211611 26319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:29.213414 26319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:29.213502 26319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:29.213541 26319 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:29.213580 26319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:29.215133 26319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:42577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:29.215444 26319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:29.215713 26319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:29.221201 26324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:29.221230 26325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:29.221217 26328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:29.221692 26319 server_base.cc:1034] running on GCE node
I20260810 07:24:29.221979 26319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:29.222249 26319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:29.223404 26319 hybrid_clock.cc:648] HybridClock initialized: now 1786346669223382 us; error 39 us; skew 500 ppm
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:29.226189 26319 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:29.226621 26319 webserver.cc:460] Webserver started at http://127.24.60.194:35011/ using document root <none> and password file <none>
I20260810 07:24:29.226814 26319 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:29.226856 26319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:29.226948 26319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:29.228119 26319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "37b690dc5cea45a99697765648bae379"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "66adfb325296cdc8f9fe2c498e385660"
server_key_iv: "330d4325dfd033a211be12c8c72d0094"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.228430 26319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "37b690dc5cea45a99697765648bae379"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "66adfb325296cdc8f9fe2c498e385660"
server_key_iv: "330d4325dfd033a211be12c8c72d0094"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.229909 26319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:29.230700 26336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.230912 26319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:29.231014 26319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "37b690dc5cea45a99697765648bae379"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "66adfb325296cdc8f9fe2c498e385660"
server_key_iv: "330d4325dfd033a211be12c8c72d0094"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.231089 26319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:29.243857 26319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:29.246668 26319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:29.246831 26319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:29.247467 26319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:29.247514 26319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.247537 26319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:29.247555 26319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:29.255539 26319 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:43473
I20260810 07:24:29.255586 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:43473 every 8 connection(s)
I20260810 07:24:29.255937 26319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:29.258984 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.248164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:38779 (local address 127.24.60.254:42577)
0810 07:24:29.248223 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.248225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.248472 (+ 247us) server_negotiation.cc:407] Connection header received
0810 07:24:29.249301 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.249302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.249319 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.249409 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.250247 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.250807 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.252046 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.252416 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.252518 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.255741 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.255743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.255743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.255752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.257500 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.257947 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.257949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.257949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.258016 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.258280 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.258281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.258282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.258423 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.258500 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.258798 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.258885 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:24:29.259204 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.248114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.24.60.194:38779)
0810 07:24:29.248332 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.248374 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.249113 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.249534 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.249538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.249820 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.250045 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.250051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.250926 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.250927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.251929 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.251933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.252551 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.252553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.252739 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.253510 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.253521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.255641 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.257611 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.257620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.257622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.257864 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.258087 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.258088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.258089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.258209 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.258505 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.258514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.258664 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.258777 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.258958 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":102,"threads_started":1}
I20260810 07:24:29.259780 26472 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42577
I20260810 07:24:29.259871 26472 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:29.260066 26472 heartbeater.cc:511] Master 127.24.60.254:42577 requested a full tablet report, sending...
I20260810 07:24:29.260495 26079 ts_manager.cc:194] Registered new tserver with Master: 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473)
I20260810 07:24:29.260898 26079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:38779
I20260810 07:24:29.263959 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 26319
I20260810 07:24:29.264068 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:29.264266 24819 external_mini_cluster.cc:1428] Setting key 4c87d11878bce7e2d3d40663a4127c4a
I20260810 07:24:29.267995 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.261304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:52789 (local address 127.24.60.254:42577)
0810 07:24:29.261354 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.261356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.261497 (+ 141us) server_negotiation.cc:407] Connection header received
0810 07:24:29.261555 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.261556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.261564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.261607 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.262035 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.262527 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.263326 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.263524 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.263603 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.264886 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.264887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.264887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.264894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.266380 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.266838 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.266840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.266840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.267031 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.267329 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.267330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.267331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.267518 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.267606 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.267788 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.267884 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:29.268187 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.261263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.24.60.194:52789)
0810 07:24:29.261417 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.261429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.261496 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.261622 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.261623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.261839 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.261913 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.261916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.262649 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.262650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.263220 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.263223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.263605 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.263606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.263668 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.264444 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.264449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.264758 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.266494 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.266496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.266496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.266752 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.267132 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.267133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.267133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.267215 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.267597 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.267598 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.267652 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.267744 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.267939 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:29.292613 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42577
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:29.381646 26483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:29.381984 26483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:29.382027 26483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:29.384683 26483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:29.384789 26483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:29.384835 26483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:29.384872 26483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:29.387823 26483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:42577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:29.388572 26483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:29.388932 26483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:29.394995 26490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:29.395018 26494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:29.395267 26483 server_base.cc:1034] running on GCE node
W20260810 07:24:29.395467 26491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:29.395716 26483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:29.396021 26483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:29.397217 26483 hybrid_clock.cc:648] HybridClock initialized: now 1786346669397195 us; error 43 us; skew 500 ppm
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:29.400135 26483 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:29.400648 26483 webserver.cc:460] Webserver started at http://127.24.60.195:42159/ using document root <none> and password file <none>
I20260810 07:24:29.400920 26483 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:29.400996 26483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:29.401201 26483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:29.402355 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "573afc859d9840918d6b1fcf3bb8fe70"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "dc29e8f7142c3d79762128289895eb9b"
server_key_iv: "6e31aa3807d613d449f6433f415ab5d9"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.402685 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "573afc859d9840918d6b1fcf3bb8fe70"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "dc29e8f7142c3d79762128289895eb9b"
server_key_iv: "6e31aa3807d613d449f6433f415ab5d9"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.404122 26483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:29.405085 26501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.405361 26483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:29.405464 26483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "573afc859d9840918d6b1fcf3bb8fe70"
format_stamp: "Formatted at 2026-08-10 07:24:29 on dist-test-slave-vxj2"
server_key: "dc29e8f7142c3d79762128289895eb9b"
server_key_iv: "6e31aa3807d613d449f6433f415ab5d9"
server_key_version: "encryptionkey@0"
I20260810 07:24:29.405536 26483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:29.416980 26483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:29.419376 26483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:29.419508 26483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:29.420056 26483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:29.420101 26483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.420127 26483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:29.420140 26483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.428037 26483 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:43497
I20260810 07:24:29.428170 26627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:43497 every 8 connection(s)
I20260810 07:24:29.428478 26483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:29.428900 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 26483
I20260810 07:24:29.429003 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:29.429231 24819 external_mini_cluster.cc:1428] Setting key f603c2dd3e0617535c0b0202b2bfc1b1
I20260810 07:24:29.432703 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.420649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:50185 (local address 127.24.60.254:42577)
0810 07:24:29.420699 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.420701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.421032 (+ 331us) server_negotiation.cc:407] Connection header received
0810 07:24:29.421744 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.421746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.421763 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.421846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.422908 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.423815 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.425788 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.426106 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.426216 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.429534 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.429535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.429535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.429546 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.431233 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.431679 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.431680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.431680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.431741 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.432071 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.432072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.432073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.432196 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.432250 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.432562 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.432637 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 07:24:29.432915 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.420660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.24.60.195:50185)
0810 07:24:29.420839 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.420879 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.421577 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.422016 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.422022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.422459 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.422786 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.422795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.423954 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.423955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.425622 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.425627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.426250 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.426252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.426428 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.427251 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.427271 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.429402 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.431336 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.431344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.431346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.431595 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.431851 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.431852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.431853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.431988 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.432280 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.432287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.432429 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.432519 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.432659 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260810 07:24:29.433506 26628 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42577
I20260810 07:24:29.433600 26628 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:29.433890 26628 heartbeater.cc:511] Master 127.24.60.254:42577 requested a full tablet report, sending...
I20260810 07:24:29.434324 26078 ts_manager.cc:194] Registered new tserver with Master: 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497)
I20260810 07:24:29.434933 26078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:50185
I20260810 07:24:29.440680 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.434705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:48089 (local address 127.24.60.254:42577)
0810 07:24:29.434743 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.434745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.434924 (+ 179us) server_negotiation.cc:407] Connection header received
0810 07:24:29.435011 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.435012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.435023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.435082 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.435529 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.436073 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.436876 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.437100 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.437229 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.438290 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.438291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.438291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.438298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.439619 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.440027 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.440028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.440028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.440081 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.440259 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.440260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.440260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.440359 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.440407 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.440538 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.440617 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 07:24:29.440765 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.434653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.24.60.195:48089)
0810 07:24:29.434852 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.434863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.434940 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.435130 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.435131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.435379 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.435444 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.435447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.436172 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.436173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.436759 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.436761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.437230 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.437231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.437312 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.437891 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.437895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.438202 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.439703 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.439703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.439703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.439944 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.440143 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.440144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.440144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.440203 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.440418 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.440419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.440463 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.440520 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.440647 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":129,"threads_started":1}
I20260810 07:24:29.441510 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:29.468987 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.460316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.24.60.254:42577)
0810 07:24:29.460387 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.460389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.460399 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:29.460429 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.460430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.460442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.460510 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.461349 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.461907 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.462706 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.462923 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.462998 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.465578 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.465580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.465580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.465588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.467300 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.467878 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.467879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.467880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.467965 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.468248 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.468249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.468249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.468377 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.468472 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.468687 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.468766 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:24:29.471078 26078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:29.472867 26078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:29.481083 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.476237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43473 (local address 127.0.0.1:52754)
0810 07:24:29.476581 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.476635 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.476714 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.477526 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.477527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.477529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.477585 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.477586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.478523 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.478524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.479969 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.479972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.480742 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.480825 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.480841 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.480915 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.480979 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":78,"thread_start_us":112,"threads_started":1}
I20260810 07:24:29.481093 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.476386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.24.60.194:43473)
0810 07:24:29.477325 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.477329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.477339 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:29.477378 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.477380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.477401 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.477663 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.477732 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.478410 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480077 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.480643 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480742 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.480811 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.480955 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.480991 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":105,"threads_started":1}
I20260810 07:24:29.481554 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.476380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.24.60.193:39437)
0810 07:24:29.476830 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.476835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.476846 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:29.476894 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.476896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.476910 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.477026 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.477285 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.478495 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480067 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.480798 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480865 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.480913 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.481396 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.481443 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":82,"threads_started":1}
I20260810 07:24:29.481585 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.476565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.24.60.195:43497)
0810 07:24:29.476794 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.476799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.477298 (+ 499us) server_negotiation.cc:407] Connection header received
0810 07:24:29.477359 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.477362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.477380 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.477466 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.477663 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.478720 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.479943 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.480723 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480824 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.480891 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.481432 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.481475 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":115,"threads_started":1}
I20260810 07:24:29.481732 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.476437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:43497 (local address 127.0.0.1:57606)
0810 07:24:29.477132 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.477143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.477225 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.477478 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.477479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.477481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.477570 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.477573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.478817 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.478819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.479808 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.479812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.480935 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.481013 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.481019 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.481078 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.481122 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":22,"thread_start_us":81,"threads_started":1}
I20260810 07:24:29.481953 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.476183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:39437 (local address 127.0.0.1:49082)
0810 07:24:29.476580 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.476631 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.476711 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.477032 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.477035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.477038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.477126 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.477129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.478589 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.478589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.479969 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.479972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.480946 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.480947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.481024 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.481027 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.481077 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.481122 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":37,"thread_start_us":92,"threads_started":1}
I20260810 07:24:29.482183 26395 tablet_service.cc:1467] Processing CreateTablet for tablet e578293ca897451db156e228c1aef924 (DEFAULT_TABLE table=test-table [id=6f78033d9d6546059f920838677109c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:29.482576 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e578293ca897451db156e228c1aef924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.485249 26233 tablet_service.cc:1467] Processing CreateTablet for tablet e578293ca897451db156e228c1aef924 (DEFAULT_TABLE table=test-table [id=6f78033d9d6546059f920838677109c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:29.485249 26557 tablet_service.cc:1467] Processing CreateTablet for tablet e578293ca897451db156e228c1aef924 (DEFAULT_TABLE table=test-table [id=6f78033d9d6546059f920838677109c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:29.485677 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e578293ca897451db156e228c1aef924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.485677 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e578293ca897451db156e228c1aef924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.486203 26656 tablet_bootstrap.cc:492] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Bootstrap starting.
I20260810 07:24:29.486873 26656 tablet_bootstrap.cc:654] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.487234 26656 log.cc:824] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:29.488246 26656 tablet_bootstrap.cc:492] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: No bootstrap required, opened a new log
I20260810 07:24:29.488330 26656 ts_tablet_manager.cc:1397] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:24:29.489241 26659 tablet_bootstrap.cc:492] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Bootstrap starting.
I20260810 07:24:29.489588 26660 tablet_bootstrap.cc:492] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Bootstrap starting.
I20260810 07:24:29.490177 26660 tablet_bootstrap.cc:654] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.490496 26659 tablet_bootstrap.cc:654] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.490514 26660 log.cc:824] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:29.490685 26656 raft_consensus.cc:348] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.490842 26659 log.cc:824] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:29.490876 26656 raft_consensus.cc:374] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.490914 26656 raft_consensus.cc:729] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b690dc5cea45a99697765648bae379, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.491101 26656 consensus_queue.cc:260] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.491348 26660 tablet_bootstrap.cc:492] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: No bootstrap required, opened a new log
I20260810 07:24:29.491400 26656 ts_tablet_manager.cc:1428] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:29.491426 26660 ts_tablet_manager.cc:1397] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:29.491492 26472 heartbeater.cc:503] Master 127.24.60.254:42577 was elected leader, sending a full tablet report...
I20260810 07:24:29.492072 26659 tablet_bootstrap.cc:492] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: No bootstrap required, opened a new log
I20260810 07:24:29.492146 26659 ts_tablet_manager.cc:1397] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:24:29.493532 26660 raft_consensus.cc:348] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.493705 26660 raft_consensus.cc:374] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.493739 26660 raft_consensus.cc:729] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4995a6dab4bc4bbd89f0bdcdf143d33e, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.493862 26660 consensus_queue.cc:260] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.494141 26660 ts_tablet_manager.cc:1428] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 07:24:29.494212 26307 heartbeater.cc:503] Master 127.24.60.254:42577 was elected leader, sending a full tablet report...
I20260810 07:24:29.494257 26659 raft_consensus.cc:348] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.494436 26659 raft_consensus.cc:374] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.494460 26659 raft_consensus.cc:729] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573afc859d9840918d6b1fcf3bb8fe70, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.494565 26659 consensus_queue.cc:260] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.494833 26659 ts_tablet_manager.cc:1428] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:29.494866 26628 heartbeater.cc:503] Master 127.24.60.254:42577 was elected leader, sending a full tablet report...
W20260810 07:24:29.506635 26473 tablet.cc:2369] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:24:29.583611 26309 tablet.cc:2369] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:29.633733 26665 raft_consensus.cc:482] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:29.633931 26665 raft_consensus.cc:504] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.634387 26665 leader_election.cc:288] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473), 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497)
I20260810 07:24:29.639068 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.634662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:47747 (local address 127.24.60.195:43497)
0810 07:24:29.634707 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.634710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.634810 (+ 100us) server_negotiation.cc:407] Connection header received
0810 07:24:29.634862 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.634863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.634875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.634944 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.635276 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.636047 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.637912 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.638514 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.638611 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.638625 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.638937 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.638987 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:29.639068 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.634623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58479 (local address 127.24.60.194:43473)
0810 07:24:29.634707 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.634710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.634817 (+ 107us) server_negotiation.cc:407] Connection header received
0810 07:24:29.634886 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.634887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.634896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.634950 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.635276 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.636107 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.637912 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.638525 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.638617 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.638626 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.638936 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.638987 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:24:29.639114 26668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.634552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:43497 (local address 127.24.60.193:47747)
0810 07:24:29.634707 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.634720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.634802 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.634968 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.634990 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.634993 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.635134 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.635136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.636184 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.636185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.637788 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.637793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.638687 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.638689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.638821 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.638831 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.638909 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.638976 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":114,"thread_start_us":80,"threads_started":1}
I20260810 07:24:29.639119 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.634552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43473 (local address 127.24.60.193:58479)
0810 07:24:29.634737 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.634747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.634805 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.635021 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.635021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.635023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.635118 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.635121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.636199 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.636200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.637789 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.637793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.638689 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.638690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.638821 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.638830 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.638909 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.638976 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":117,"thread_start_us":67,"threads_started":1}
I20260810 07:24:29.639523 26581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e578293ca897451db156e228c1aef924" candidate_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573afc859d9840918d6b1fcf3bb8fe70" is_pre_election: true
I20260810 07:24:29.639732 26581 raft_consensus.cc:2454] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4995a6dab4bc4bbd89f0bdcdf143d33e in term 0.
I20260810 07:24:29.640076 26180 leader_election.cc:302] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4995a6dab4bc4bbd89f0bdcdf143d33e, 573afc859d9840918d6b1fcf3bb8fe70; no voters:
I20260810 07:24:29.640223 26665 raft_consensus.cc:2781] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:29.640189 26421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e578293ca897451db156e228c1aef924" candidate_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b690dc5cea45a99697765648bae379" is_pre_election: true
I20260810 07:24:29.640290 26665 raft_consensus.cc:482] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:29.640319 26665 raft_consensus.cc:3037] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.640393 26421 raft_consensus.cc:2454] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4995a6dab4bc4bbd89f0bdcdf143d33e in term 0.
I20260810 07:24:29.641211 26665 raft_consensus.cc:504] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.641386 26665 leader_election.cc:288] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [CANDIDATE]: Term 1 election: Requested vote from peers 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473), 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497)
I20260810 07:24:29.641594 26421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e578293ca897451db156e228c1aef924" candidate_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b690dc5cea45a99697765648bae379"
I20260810 07:24:29.641595 26581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e578293ca897451db156e228c1aef924" candidate_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573afc859d9840918d6b1fcf3bb8fe70"
I20260810 07:24:29.641696 26421 raft_consensus.cc:3037] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.641696 26581 raft_consensus.cc:3037] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.642719 26421 raft_consensus.cc:2454] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4995a6dab4bc4bbd89f0bdcdf143d33e in term 1.
I20260810 07:24:29.642719 26581 raft_consensus.cc:2454] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4995a6dab4bc4bbd89f0bdcdf143d33e in term 1.
I20260810 07:24:29.643062 26178 leader_election.cc:302] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b690dc5cea45a99697765648bae379, 4995a6dab4bc4bbd89f0bdcdf143d33e; no voters:
I20260810 07:24:29.643246 26665 raft_consensus.cc:2781] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:29.643458 26665 raft_consensus.cc:686] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 LEADER]: Becoming Leader. State: Replica: 4995a6dab4bc4bbd89f0bdcdf143d33e, State: Running, Role: LEADER
I20260810 07:24:29.643590 26665 consensus_queue.cc:237] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.644611 26079 catalog_manager.cc:5495] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193). New cstate: current_term: 1 leader_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:29.665849 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.663160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.24.60.193:39437)
0810 07:24:29.663235 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.663237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.663247 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:29.663334 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.663335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.663355 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.663423 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:29.663561 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.664082 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.664875 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.665142 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.665282 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.665483 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.665588 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.665655 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.665702 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.665723 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:24:29.668252 26421 raft_consensus.cc:1261] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Refusing update from remote peer 4995a6dab4bc4bbd89f0bdcdf143d33e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:29.668253 26581 raft_consensus.cc:1261] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Refusing update from remote peer 4995a6dab4bc4bbd89f0bdcdf143d33e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:29.668656 26669 consensus_queue.cc:1035] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [LEADER]: Connected to new peer: Peer: permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:29.668785 26665 consensus_queue.cc:1035] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:29.672348 26675 mvcc.cc:205] Tried to move back new op lower bound from 7316875958957473792 to 7316875958860804096. Current Snapshot: MvccSnapshot[applied={T|T < 7316875958957473792}]
I20260810 07:24:29.675792 26078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:29.675920 26078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:29.676806 26078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: Sending DeleteTablet for 3 replicas of tablet e578293ca897451db156e228c1aef924
I20260810 07:24:29.677266 26395 tablet_service.cc:1514] Processing DeleteTablet for tablet e578293ca897451db156e228c1aef924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:29 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:52754
I20260810 07:24:29.677264 26557 tablet_service.cc:1514] Processing DeleteTablet for tablet e578293ca897451db156e228c1aef924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:29 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:57606
I20260810 07:24:29.677280 26233 tablet_service.cc:1514] Processing DeleteTablet for tablet e578293ca897451db156e228c1aef924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:29 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49082
I20260810 07:24:29.677619 26686 tablet_replica.cc:330] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: stopping tablet replica
I20260810 07:24:29.677729 26686 raft_consensus.cc:2229] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:29.677886 26686 raft_consensus.cc:2258] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:29.677922 26078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:29.678088 26078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:29.678082 26687 tablet_replica.cc:330] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: stopping tablet replica
I20260810 07:24:29.678184 26687 raft_consensus.cc:2229] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:29.678314 26687 raft_consensus.cc:2258] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:29.678651 26688 tablet_replica.cc:330] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: stopping tablet replica
I20260810 07:24:29.678771 26688 raft_consensus.cc:2229] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:29.679162 26688 raft_consensus.cc:2258] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:29.679646 26233 tablet_service.cc:1467] Processing CreateTablet for tablet e97b2983b298429a879d7e5da472f666 (DEFAULT_TABLE table=test-table [id=6e75abba4ac74bf2b918eb22b6891375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:29.679790 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97b2983b298429a879d7e5da472f666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.679848 26395 tablet_service.cc:1467] Processing CreateTablet for tablet e97b2983b298429a879d7e5da472f666 (DEFAULT_TABLE table=test-table [id=6e75abba4ac74bf2b918eb22b6891375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:29.679981 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97b2983b298429a879d7e5da472f666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.681113 26660 tablet_bootstrap.cc:492] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Bootstrap starting.
I20260810 07:24:29.681408 26656 tablet_bootstrap.cc:492] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Bootstrap starting.
I20260810 07:24:29.681525 26686 ts_tablet_manager.cc:1905] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:29.681604 26660 tablet_bootstrap.cc:654] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.681766 26656 tablet_bootstrap.cc:654] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.682703 26660 tablet_bootstrap.cc:492] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: No bootstrap required, opened a new log
I20260810 07:24:29.682757 26660 ts_tablet_manager.cc:1397] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:29.682925 26660 raft_consensus.cc:348] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.683008 26660 raft_consensus.cc:374] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.683032 26660 raft_consensus.cc:729] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4995a6dab4bc4bbd89f0bdcdf143d33e, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.683081 26660 consensus_queue.cc:260] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.683183 26660 ts_tablet_manager.cc:1428] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.679561 26687 ts_tablet_manager.cc:1905] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:29.680140 26557 tablet_service.cc:1467] Processing CreateTablet for tablet e97b2983b298429a879d7e5da472f666 (DEFAULT_TABLE table=test-table [id=6e75abba4ac74bf2b918eb22b6891375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:29.683451 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97b2983b298429a879d7e5da472f666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.683928 26686 ts_tablet_manager.cc:1918] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:29.684010 26686 log.cc:1196] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/e578293ca897451db156e228c1aef924
I20260810 07:24:29.684288 26686 ts_tablet_manager.cc:1939] T e578293ca897451db156e228c1aef924 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Deleting consensus metadata
I20260810 07:24:29.684775 26659 tablet_bootstrap.cc:492] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Bootstrap starting.
I20260810 07:24:29.685204 26688 ts_tablet_manager.cc:1905] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:29.685258 26659 tablet_bootstrap.cc:654] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.685834 26064 catalog_manager.cc:4841] TS 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437): tablet e578293ca897451db156e228c1aef924 (table test-table [id=6f78033d9d6546059f920838677109c3]) successfully deleted
I20260810 07:24:29.686976 26659 tablet_bootstrap.cc:492] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: No bootstrap required, opened a new log
I20260810 07:24:29.687008 26687 ts_tablet_manager.cc:1918] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:29.687031 26659 ts_tablet_manager.cc:1397] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 07:24:29.687081 26687 log.cc:1196] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/e578293ca897451db156e228c1aef924
I20260810 07:24:29.687224 26688 ts_tablet_manager.cc:1918] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:29.687216 26659 raft_consensus.cc:348] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.687302 26688 log.cc:1196] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/e578293ca897451db156e228c1aef924
I20260810 07:24:29.687306 26659 raft_consensus.cc:374] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.687328 26659 raft_consensus.cc:729] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573afc859d9840918d6b1fcf3bb8fe70, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.687376 26659 consensus_queue.cc:260] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.687417 26687 ts_tablet_manager.cc:1939] T e578293ca897451db156e228c1aef924 P 573afc859d9840918d6b1fcf3bb8fe70: Deleting consensus metadata
I20260810 07:24:29.687479 26659 ts_tablet_manager.cc:1428] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.687580 26688 ts_tablet_manager.cc:1939] T e578293ca897451db156e228c1aef924 P 37b690dc5cea45a99697765648bae379: Deleting consensus metadata
I20260810 07:24:29.687911 26064 catalog_manager.cc:4841] TS 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497): tablet e578293ca897451db156e228c1aef924 (table test-table [id=6f78033d9d6546059f920838677109c3]) successfully deleted
I20260810 07:24:29.688097 26656 tablet_bootstrap.cc:492] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: No bootstrap required, opened a new log
I20260810 07:24:29.688143 26656 ts_tablet_manager.cc:1397] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 07:24:29.688313 26656 raft_consensus.cc:348] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.688324 26062 catalog_manager.cc:4841] TS 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473): tablet e578293ca897451db156e228c1aef924 (table test-table [id=6f78033d9d6546059f920838677109c3]) successfully deleted
I20260810 07:24:29.688388 26656 raft_consensus.cc:374] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.688421 26656 raft_consensus.cc:729] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b690dc5cea45a99697765648bae379, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.688472 26656 consensus_queue.cc:260] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.688606 26656 ts_tablet_manager.cc:1428] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:29.705914 26664 raft_consensus.cc:482] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:29.706048 26664 raft_consensus.cc:504] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.706410 26664 leader_election.cc:288] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437), 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497)
I20260810 07:24:29.710157 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.706662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:39437 (local address 127.24.60.194:43631)
0810 07:24:29.706734 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.706748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.706824 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.707077 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.707078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.707080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.707166 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.707168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.708009 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.708010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.709045 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.709048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.709862 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.709863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.709956 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.709961 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.710027 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.710071 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 07:24:29.710170 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.706728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:43631 (local address 127.24.60.193:39437)
0810 07:24:29.706804 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.706806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.706819 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:24:29.706925 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.706926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.706938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.707015 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.707287 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.707930 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.709175 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.709766 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.709852 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.709863 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.710046 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.710089 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:24:29.710697 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.706801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:43497 (local address 127.24.60.194:42367)
0810 07:24:29.706956 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.706967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.707037 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.707248 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.707250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.707252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.707332 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.707334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.708276 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.708277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.709518 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.709522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.710407 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.710408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.710487 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.710491 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.710576 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.710623 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260810 07:24:29.710616 26254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97b2983b298429a879d7e5da472f666" candidate_uuid: "37b690dc5cea45a99697765648bae379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" is_pre_election: true
I20260810 07:24:29.710719 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.707070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:42367 (local address 127.24.60.195:43497)
0810 07:24:29.707125 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.707127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.707135 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:29.707161 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.707162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.707171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.707224 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.707410 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.708170 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.709615 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.710208 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.710290 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.710301 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.710599 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.710653 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 07:24:29.710794 26254 raft_consensus.cc:2454] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37b690dc5cea45a99697765648bae379 in term 0.
I20260810 07:24:29.710965 26581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97b2983b298429a879d7e5da472f666" candidate_uuid: "37b690dc5cea45a99697765648bae379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573afc859d9840918d6b1fcf3bb8fe70" is_pre_election: true
I20260810 07:24:29.711017 26341 leader_election.cc:302] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b690dc5cea45a99697765648bae379, 4995a6dab4bc4bbd89f0bdcdf143d33e; no voters:
I20260810 07:24:29.711067 26581 raft_consensus.cc:2454] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37b690dc5cea45a99697765648bae379 in term 0.
I20260810 07:24:29.711150 26664 raft_consensus.cc:2781] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:29.711180 26664 raft_consensus.cc:482] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:29.711201 26664 raft_consensus.cc:3037] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.712107 26664 raft_consensus.cc:504] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.712248 26664 leader_election.cc:288] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [CANDIDATE]: Term 1 election: Requested vote from peers 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437), 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497)
I20260810 07:24:29.712555 26254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97b2983b298429a879d7e5da472f666" candidate_uuid: "37b690dc5cea45a99697765648bae379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e"
I20260810 07:24:29.712548 26581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97b2983b298429a879d7e5da472f666" candidate_uuid: "37b690dc5cea45a99697765648bae379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573afc859d9840918d6b1fcf3bb8fe70"
I20260810 07:24:29.712642 26581 raft_consensus.cc:3037] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.712642 26254 raft_consensus.cc:3037] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.713586 26581 raft_consensus.cc:2454] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37b690dc5cea45a99697765648bae379 in term 1.
I20260810 07:24:29.713647 26254 raft_consensus.cc:2454] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37b690dc5cea45a99697765648bae379 in term 1.
I20260810 07:24:29.713819 26341 leader_election.cc:302] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b690dc5cea45a99697765648bae379, 573afc859d9840918d6b1fcf3bb8fe70; no voters:
I20260810 07:24:29.713963 26664 raft_consensus.cc:2781] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:29.714129 26664 raft_consensus.cc:686] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 1 LEADER]: Becoming Leader. State: Replica: 37b690dc5cea45a99697765648bae379, State: Running, Role: LEADER
I20260810 07:24:29.714201 26664 consensus_queue.cc:237] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.714970 26078 catalog_manager.cc:5495] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37b690dc5cea45a99697765648bae379 (127.24.60.194). New cstate: current_term: 1 leader_uuid: "37b690dc5cea45a99697765648bae379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } health_report { overall_health: UNKNOWN } } }
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346668, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:29.746418 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.738038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.24.60.254:42577)
0810 07:24:29.738095 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.738097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.738229 (+ 132us) server_negotiation.cc:407] Connection header received
0810 07:24:29.738294 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.738295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.738304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.738359 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.738976 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.739515 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.740478 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.740675 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.740730 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.743373 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.743374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.743375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.743384 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.744979 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.745480 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.745481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.745482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.745546 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.745766 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.745767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.745768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.745893 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.745979 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.746157 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.746278 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 07:24:29.746507 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.737945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42577 (local address 127.0.0.1:49630)
0810 07:24:29.738142 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.738156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.738232 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.738388 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.738389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.738813 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:29.738876 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.738879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.739601 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.739602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.740394 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.740396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.740757 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.740758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.740817 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.741473 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:29.741479 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:29.743276 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:29.745090 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.745099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.745101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.745376 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.745622 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:29.745623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:29.745623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:29.745690 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:29.745968 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:29.745978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:29.746031 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.746128 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.746349 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":101,"thread_start_us":104,"threads_started":1}
I20260810 07:24:29.748814 26078 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49630:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 07:24:29.749028 26078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:29.750666 26233 tablet_service.cc:1467] Processing CreateTablet for tablet b3ee2426793a49849cd17b3653d399fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f71c2187ca654940a6ffa070537d652f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:29.750857 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ee2426793a49849cd17b3653d399fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.750810 26557 tablet_service.cc:1467] Processing CreateTablet for tablet b3ee2426793a49849cd17b3653d399fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f71c2187ca654940a6ffa070537d652f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:29.750975 26557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ee2426793a49849cd17b3653d399fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.752511 26660 tablet_bootstrap.cc:492] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: Bootstrap starting.
I20260810 07:24:29.752583 26395 tablet_service.cc:1467] Processing CreateTablet for tablet b3ee2426793a49849cd17b3653d399fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f71c2187ca654940a6ffa070537d652f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:29.752748 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ee2426793a49849cd17b3653d399fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:29.753031 26660 tablet_bootstrap.cc:654] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.753291 26659 tablet_bootstrap.cc:492] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: Bootstrap starting.
I20260810 07:24:29.753726 26659 tablet_bootstrap.cc:654] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.754601 26656 tablet_bootstrap.cc:492] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: Bootstrap starting.
I20260810 07:24:29.754822 26659 tablet_bootstrap.cc:492] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: No bootstrap required, opened a new log
I20260810 07:24:29.754869 26659 ts_tablet_manager.cc:1397] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 07:24:29.755100 26656 tablet_bootstrap.cc:654] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:29.755060 26659 raft_consensus.cc:348] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.755160 26659 raft_consensus.cc:374] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.755188 26659 raft_consensus.cc:729] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573afc859d9840918d6b1fcf3bb8fe70, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.755263 26659 consensus_queue.cc:260] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.755384 26659 ts_tablet_manager.cc:1428] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.755543 26666 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.755631 26666 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.755942 26656 tablet_bootstrap.cc:492] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: No bootstrap required, opened a new log
I20260810 07:24:29.755998 26656 ts_tablet_manager.cc:1397] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:29.756209 26660 tablet_bootstrap.cc:492] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: No bootstrap required, opened a new log
I20260810 07:24:29.756172 26656 raft_consensus.cc:348] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.756247 26656 raft_consensus.cc:374] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.756256 26660 ts_tablet_manager.cc:1397] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260810 07:24:29.756276 26656 raft_consensus.cc:729] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b690dc5cea45a99697765648bae379, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.756327 26656 consensus_queue.cc:260] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.756449 26656 ts_tablet_manager.cc:1428] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:29.756436 26660 raft_consensus.cc:348] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.756518 26660 raft_consensus.cc:374] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:29.756543 26660 raft_consensus.cc:729] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4995a6dab4bc4bbd89f0bdcdf143d33e, State: Initialized, Role: FOLLOWER
I20260810 07:24:29.756496 26664 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.756597 26664 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.756593 26660 consensus_queue.cc:260] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.756736 26660 ts_tablet_manager.cc:1428] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 07:24:29.756861 26679 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.756968 26679 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.814870 26666 raft_consensus.cc:482] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:29.815024 26666 raft_consensus.cc:504] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.815505 26666 leader_election.cc:288] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437), 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473)
I20260810 07:24:29.819747 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.815687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43473 (local address 127.24.60.195:41543)
0810 07:24:29.815746 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.815760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.815804 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.816037 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.816038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.816041 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.816154 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.816157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.817189 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.817190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.818818 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.818822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.819529 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.819529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.819595 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.819599 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.819647 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.819681 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":80}
I20260810 07:24:29.819800 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.815750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:41543 (local address 127.24.60.194:43473)
0810 07:24:29.815801 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.815803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.815814 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:29.815921 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.815922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.815932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.816020 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.816255 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.817071 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.818924 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.819435 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.819515 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.819525 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.819678 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.819724 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:24:29.820097 26421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee2426793a49849cd17b3653d399fb" candidate_uuid: "573afc859d9840918d6b1fcf3bb8fe70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b690dc5cea45a99697765648bae379" is_pre_election: true
I20260810 07:24:29.820170 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.815936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:50217 (local address 127.24.60.193:39437)
0810 07:24:29.815979 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.815982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.815995 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:24:29.816084 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.816085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.816096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.816146 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:29.816340 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.817042 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.819197 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.819774 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.819846 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.819856 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.820049 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.820104 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:24:29.820158 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.815705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:39437 (local address 127.24.60.195:50217)
0810 07:24:29.815946 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.815956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.816020 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.816178 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.816179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.816180 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:29.816259 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.816260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.817199 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.817199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.819102 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.819104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.819896 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.819897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.819982 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.819987 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.820035 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.820073 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":89,"thread_start_us":152,"threads_started":1}
I20260810 07:24:29.820215 26421 raft_consensus.cc:2454] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573afc859d9840918d6b1fcf3bb8fe70 in term 0.
I20260810 07:24:29.820412 26254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee2426793a49849cd17b3653d399fb" candidate_uuid: "573afc859d9840918d6b1fcf3bb8fe70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" is_pre_election: true
I20260810 07:24:29.820518 26254 raft_consensus.cc:2454] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573afc859d9840918d6b1fcf3bb8fe70 in term 0.
I20260810 07:24:29.820465 26503 leader_election.cc:302] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b690dc5cea45a99697765648bae379, 573afc859d9840918d6b1fcf3bb8fe70; no voters:
I20260810 07:24:29.820633 26666 raft_consensus.cc:2781] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:29.820681 26666 raft_consensus.cc:482] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:29.820708 26666 raft_consensus.cc:3037] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.821434 26666 raft_consensus.cc:504] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.821606 26666 leader_election.cc:288] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [CANDIDATE]: Term 1 election: Requested vote from peers 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437), 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473)
I20260810 07:24:29.821812 26254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee2426793a49849cd17b3653d399fb" candidate_uuid: "573afc859d9840918d6b1fcf3bb8fe70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e"
I20260810 07:24:29.821805 26421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ee2426793a49849cd17b3653d399fb" candidate_uuid: "573afc859d9840918d6b1fcf3bb8fe70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b690dc5cea45a99697765648bae379"
I20260810 07:24:29.821905 26254 raft_consensus.cc:3037] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.821903 26421 raft_consensus.cc:3037] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:29.822638 26254 raft_consensus.cc:2454] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573afc859d9840918d6b1fcf3bb8fe70 in term 1.
I20260810 07:24:29.822873 26505 leader_election.cc:302] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4995a6dab4bc4bbd89f0bdcdf143d33e, 573afc859d9840918d6b1fcf3bb8fe70; no voters:
I20260810 07:24:29.822880 26421 raft_consensus.cc:2454] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573afc859d9840918d6b1fcf3bb8fe70 in term 1.
I20260810 07:24:29.822974 26666 raft_consensus.cc:2781] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:29.823149 26666 raft_consensus.cc:686] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 LEADER]: Becoming Leader. State: Replica: 573afc859d9840918d6b1fcf3bb8fe70, State: Running, Role: LEADER
I20260810 07:24:29.823212 26666 consensus_queue.cc:237] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } }
I20260810 07:24:29.823686 26707 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: TxnStatusTablet state changed. Reason: New leader 573afc859d9840918d6b1fcf3bb8fe70. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.823789 26707 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:29.824054 26709 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 07:24:29.824144 26709 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 07:24:29.824163 26079 catalog_manager.cc:5495] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195). New cstate: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } health_report { overall_health: HEALTHY } } }
I20260810 07:24:29.836093 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.832993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.24.60.195:43497)
0810 07:24:29.833038 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.833040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.833076 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:24:29.833233 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.833234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.833272 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.833330 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:29.833529 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.834048 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.835289 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.835491 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.835548 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.835770 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.835864 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.835919 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.836011 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.836030 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 07:24:29.836166 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.832925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:43497 (local address 127.0.0.1:57620)
0810 07:24:29.833013 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.833028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.833102 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.833382 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.833383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.833385 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:29.833452 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.833455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.834147 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.834149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.835204 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.835207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.835574 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.835575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.835652 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.835672 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.835936 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.835947 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.836010 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.836073 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 07:24:29.837598 26421 raft_consensus.cc:1261] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Refusing update from remote peer 573afc859d9840918d6b1fcf3bb8fe70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:29.837620 26254 raft_consensus.cc:1261] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Refusing update from remote peer 573afc859d9840918d6b1fcf3bb8fe70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:29.837958 26707 consensus_queue.cc:1035] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:29.838130 26666 consensus_queue.cc:1035] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:29.838876 26679 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: TxnStatusTablet state changed. Reason: New leader 573afc859d9840918d6b1fcf3bb8fe70. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.838960 26679 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.838851 26664 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: TxnStatusTablet state changed. Reason: New leader 573afc859d9840918d6b1fcf3bb8fe70. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.839013 26664 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.839545 26707 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.839637 26707 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:29.839648 26674 mvcc.cc:205] Tried to move back new op lower bound from 7316875959652200448 to 7316875959596576768. Current Snapshot: MvccSnapshot[applied={T|T < 7316875959652200448}]
I20260810 07:24:29.839731 26664 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.839795 26664 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 37b690dc5cea45a99697765648bae379: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.840279 26707 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.840369 26707 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 573afc859d9840918d6b1fcf3bb8fe70: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:29.840860 26673 mvcc.cc:205] Tried to move back new op lower bound from 7316875959652200448 to 7316875959596576768. Current Snapshot: MvccSnapshot[applied={T|T < 7316875959652200448 or (T in {7316875959652200448})}]
I20260810 07:24:29.841060 26679 tablet_replica.cc:439] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573afc859d9840918d6b1fcf3bb8fe70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 } } peers { permanent_uuid: "37b690dc5cea45a99697765648bae379" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43473 } } peers { permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 } } }
I20260810 07:24:29.841138 26679 tablet_replica.cc:442] T b3ee2426793a49849cd17b3653d399fb P 4995a6dab4bc4bbd89f0bdcdf143d33e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:29.844585 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.841456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.24.60.194:43473)
0810 07:24:29.841494 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.841496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.841504 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:29.841544 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.841545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.841589 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.841671 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:29.841886 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.842374 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.843622 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.843902 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.843969 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.844183 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.844317 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.844380 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.844464 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.844485 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 07:24:29.850124 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.847518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:60641 (local address 127.24.60.195:43497)
0810 07:24:29.847573 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.847575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.847687 (+ 112us) server_negotiation.cc:407] Connection header received
0810 07:24:29.847786 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.847787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.847798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.847850 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:29.848024 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.848504 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.849350 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.849557 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.849643 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.849859 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.849898 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.849961 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.850023 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.850045 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 07:24:29.850150 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.847439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:43497 (local address 127.24.60.194:60641)
0810 07:24:29.847609 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.847620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.847707 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.847876 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.847877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.847878 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:29.847945 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.847947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.848588 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.848589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.849254 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.849256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.849673 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.849674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.849738 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.849752 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.849963 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.849966 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.850019 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.850066 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 07:24:29.852725 26254 raft_consensus.cc:1261] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Refusing update from remote peer 37b690dc5cea45a99697765648bae379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:29.852753 26581 raft_consensus.cc:1261] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Refusing update from remote peer 37b690dc5cea45a99697765648bae379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:29.853083 26693 consensus_queue.cc:1035] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4995a6dab4bc4bbd89f0bdcdf143d33e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 39437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:29.853236 26664 consensus_queue.cc:1035] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573afc859d9840918d6b1fcf3bb8fe70" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 43497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:29.868467 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.865825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:42999 (local address 127.24.60.194:43473)
0810 07:24:29.865870 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.865872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.865882 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:29.865972 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.865973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.865983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.866028 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:29.866215 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.866789 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.867670 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.867900 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.867980 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.868194 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.868223 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.868302 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.868358 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.868383 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:24:29.868476 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.865719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43473 (local address 127.24.60.195:42999)
0810 07:24:29.865821 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:29.865835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:29.865888 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:29.866058 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:29.866059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:29.866061 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:29.866143 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.866144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.866903 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.866904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.867574 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:29.867577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.867981 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.867982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:29.868055 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.868079 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:29.868307 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:29.868310 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:29.868358 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:24:29.868407 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 07:24:29.878196 26079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:29.878338 26079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:29.879063 26079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f3fe86ce3d4ed6b634a2cec87e2278: Sending DeleteTablet for 3 replicas of tablet e97b2983b298429a879d7e5da472f666
I20260810 07:24:29.879482 26395 tablet_service.cc:1514] Processing DeleteTablet for tablet e97b2983b298429a879d7e5da472f666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:29 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:52754
I20260810 07:24:29.879484 26233 tablet_service.cc:1514] Processing DeleteTablet for tablet e97b2983b298429a879d7e5da472f666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:29 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49082
I20260810 07:24:29.879587 26688 tablet_replica.cc:330] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: stopping tablet replica
I20260810 07:24:29.879635 26688 raft_consensus.cc:2229] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:29.879655 26686 tablet_replica.cc:330] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: stopping tablet replica
I20260810 07:24:29.879709 26686 raft_consensus.cc:2229] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:29.879750 26686 raft_consensus.cc:2258] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:29.879742 26688 raft_consensus.cc:2258] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:29.880173 26686 ts_tablet_manager.cc:1905] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:29.880189 26688 ts_tablet_manager.cc:1905] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:29.882221 26688 ts_tablet_manager.cc:1918] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:29.882292 26688 log.cc:1196] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/e97b2983b298429a879d7e5da472f666
I20260810 07:24:29.882532 26688 ts_tablet_manager.cc:1939] T e97b2983b298429a879d7e5da472f666 P 37b690dc5cea45a99697765648bae379: Deleting consensus metadata
I20260810 07:24:29.882680 26557 tablet_service.cc:1514] Processing DeleteTablet for tablet e97b2983b298429a879d7e5da472f666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:29 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:57606
I20260810 07:24:29.882815 26687 tablet_replica.cc:330] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: stopping tablet replica
I20260810 07:24:29.882864 26686 ts_tablet_manager.cc:1918] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:29.882920 26686 log.cc:1196] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/e97b2983b298429a879d7e5da472f666
I20260810 07:24:29.882922 26687 raft_consensus.cc:2229] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:29.882973 26687 raft_consensus.cc:2258] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:29.883136 26686 ts_tablet_manager.cc:1939] T e97b2983b298429a879d7e5da472f666 P 4995a6dab4bc4bbd89f0bdcdf143d33e: Deleting consensus metadata
I20260810 07:24:29.883533 26064 catalog_manager.cc:4841] TS 4995a6dab4bc4bbd89f0bdcdf143d33e (127.24.60.193:39437): tablet e97b2983b298429a879d7e5da472f666 (table test-table [id=6e75abba4ac74bf2b918eb22b6891375]) successfully deleted
I20260810 07:24:29.883782 26687 ts_tablet_manager.cc:1905] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:29.885020 26062 catalog_manager.cc:4841] TS 37b690dc5cea45a99697765648bae379 (127.24.60.194:43473): tablet e97b2983b298429a879d7e5da472f666 (table test-table [id=6e75abba4ac74bf2b918eb22b6891375]) successfully deleted
I20260810 07:24:29.885293 26687 ts_tablet_manager.cc:1918] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:29.885373 26687 log.cc:1196] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/e97b2983b298429a879d7e5da472f666
I20260810 07:24:29.885625 26687 ts_tablet_manager.cc:1939] T e97b2983b298429a879d7e5da472f666 P 573afc859d9840918d6b1fcf3bb8fe70: Deleting consensus metadata
I20260810 07:24:29.885995 26064 catalog_manager.cc:4841] TS 573afc859d9840918d6b1fcf3bb8fe70 (127.24.60.195:43497): tablet e97b2983b298429a879d7e5da472f666 (table test-table [id=6e75abba4ac74bf2b918eb22b6891375]) successfully deleted
Aug 10 07:24:29 dist-test-slave-vxj2 krb5kdc[26029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346669, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:29.895764 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.886126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.24.60.193:39437)
0810 07:24:29.886179 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.886181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.886459 (+ 278us) server_negotiation.cc:407] Connection header received
0810 07:24:29.886545 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.886546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.886557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.886614 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.887202 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.887809 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.888692 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.888915 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.888996 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.892380 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.892396 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.892396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.892414 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.894333 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.894766 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.894770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.894770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.894840 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.895098 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.895099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.895099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.895269 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.895441 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.895526 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.895631 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 07:24:29.896080 26301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49088
I20260810 07:24:29.904404 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:29.897206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.24.60.254:42577)
0810 07:24:29.897324 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:29.897326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:29.897620 (+ 294us) server_negotiation.cc:407] Connection header received
0810 07:24:29.897692 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:29.897693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:29.897706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:29.897761 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:29.898362 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.898928 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.899712 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:29.899956 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:29.900042 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:29.901477 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:29.901479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:29.901480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:29.901488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:29.903094 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.903504 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.903505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.903506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.903577 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:29.903807 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:29.903808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:29.903809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:29.903947 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:29.904070 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:29.904183 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 07:24:29.904280 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 07:24:29.904774 26079 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49634
I20260810 07:24:29.905656 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 26161
I20260810 07:24:29.918639 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 26319
I20260810 07:24:29.927286 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 26483
I20260810 07:24:29.936215 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 26044
2026-08-10T07:24:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2472 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:30 dist-test-slave-vxj2 krb5kdc[26734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:30 dist-test-slave-vxj2 krb5kdc[26734](info): set up 2 sockets
Aug 10 07:24:30 dist-test-slave-vxj2 krb5kdc[26734](info): commencing operation
krb5kdc: starting...
W20260810 07:24:31.112037 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:31.248605 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36735
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:31.333801 26749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:31.334067 26749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:31.334112 26749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:31.335350 26749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:31.335433 26749 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 07:24:31.335460 26749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:31.335489 26749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:31.335520 26749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:31.335543 26749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:31.337144 26749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:36735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:31.337553 26749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:31.337805 26749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:31.343286 26755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.343281 26754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.343286 26757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:31.343571 26749 server_base.cc:1034] running on GCE node
I20260810 07:24:31.343822 26749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:31.344134 26749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:31.345271 26749 hybrid_clock.cc:648] HybridClock initialized: now 1786346671345256 us; error 30 us; skew 500 ppm
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:31.354036 26749 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:31.354630 26749 webserver.cc:460] Webserver started at http://127.24.60.254:39999/ using document root <none> and password file <none>
I20260810 07:24:31.354878 26749 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:31.354954 26749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:31.355110 26749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:31.356262 26749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "2cd49c1e28dd4b5085aca4ff4d8df207"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "2ad5ef90bac290a7f4272b0d888549bc"
server_key_iv: "93e4bcf2eedab1d30c36a58f4ea7be88"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.356652 26749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "2cd49c1e28dd4b5085aca4ff4d8df207"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "2ad5ef90bac290a7f4272b0d888549bc"
server_key_iv: "93e4bcf2eedab1d30c36a58f4ea7be88"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.358222 26749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:31.359046 26764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.359282 26749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:31.359396 26749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "2cd49c1e28dd4b5085aca4ff4d8df207"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "2ad5ef90bac290a7f4272b0d888549bc"
server_key_iv: "93e4bcf2eedab1d30c36a58f4ea7be88"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.359476 26749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:31.383594 26749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:31.386860 26749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:31.387189 26749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:31.393632 26749 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:36735
I20260810 07:24:31.393695 26837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:36735 every 8 connection(s)
I20260810 07:24:31.394224 26749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:31.394668 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 26749
I20260810 07:24:31.394784 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:31.394990 24819 external_mini_cluster.cc:1428] Setting key 00ffc5ba90e8ba8dde0d0127a2af6396
I20260810 07:24:31.395134 26839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:31.399143 26839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Bootstrap starting.
I20260810 07:24:31.399919 26839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:31.400265 26839 log.cc:824] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:31.401183 26839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: No bootstrap required, opened a new log
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:31.403831 26839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36735 } }
I20260810 07:24:31.404057 26839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:31.404107 26839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd49c1e28dd4b5085aca4ff4d8df207, State: Initialized, Role: FOLLOWER
I20260810 07:24:31.404254 26839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36735 } }
I20260810 07:24:31.404343 26839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:31.404384 26839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:31.404435 26839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:31.405261 26839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36735 } }
I20260810 07:24:31.405393 26839 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd49c1e28dd4b5085aca4ff4d8df207; no voters:
I20260810 07:24:31.405629 26839 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:31.405696 26845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:31.405879 26845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd49c1e28dd4b5085aca4ff4d8df207, State: Running, Role: LEADER
I20260810 07:24:31.405939 26839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:31.405998 26845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36735 } }
I20260810 07:24:31.406376 26845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd49c1e28dd4b5085aca4ff4d8df207. Latest consensus state: current_term: 1 leader_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36735 } } }
I20260810 07:24:31.406466 26845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:31.406649 26847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd49c1e28dd4b5085aca4ff4d8df207" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36735 } } }
I20260810 07:24:31.406733 26847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:31.406857 26854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:31.407424 26854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:31.408049 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.395677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.24.60.254:36735)
0810 07:24:31.395853 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.395863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.395897 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:24:31.396550 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.396562 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.396641 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.397694 (+ 1053us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.397802 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.399132 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.400115 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.400664 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.400798 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.403391 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.403396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.403397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.403414 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.405417 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.406214 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.406217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.406217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.406340 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.406740 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.406741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.406742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.407086 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.407238 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.407683 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.407841 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260810 07:24:31.413326 26854 catalog_manager.cc:1347] Generated new cluster ID: 25cd4e3e9397491d880231ae5ec0f13c
I20260810 07:24:31.413398 26854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:31.425844 26854 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:31.426441 26854 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:31.436993 26854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Generated new TSK 0
I20260810 07:24:31.437245 26854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:31.472554 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36735
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:31.575116 26869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:31.575417 26869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:31.575467 26869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:31.576792 26869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:31.576891 26869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:31.576933 26869 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 07:24:31.576962 26869 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:31.576997 26869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:31.578760 26869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:31.579088 26869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:31.579370 26869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:31.586009 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.586009 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:31.586180 26869 server_base.cc:1034] running on GCE node
W20260810 07:24:31.586009 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:31.586606 26869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:31.586984 26869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:31.588191 26869 hybrid_clock.cc:648] HybridClock initialized: now 1786346671588164 us; error 44 us; skew 500 ppm
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:31.591933 26869 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:31.592577 26869 webserver.cc:460] Webserver started at http://127.24.60.193:34677/ using document root <none> and password file <none>
I20260810 07:24:31.592962 26869 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:31.593058 26869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:31.593273 26869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:31.594498 26869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "8ac5d35a259f4961950c3d1b85c35582"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "68da90505609247d1853f9e8c0024a5c"
server_key_iv: "32cdce2d5883a7a319301d01985be0f5"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.594926 26869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "8ac5d35a259f4961950c3d1b85c35582"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "68da90505609247d1853f9e8c0024a5c"
server_key_iv: "32cdce2d5883a7a319301d01985be0f5"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.596524 26869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:31.597471 26887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.597640 26869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:31.597757 26869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "8ac5d35a259f4961950c3d1b85c35582"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "68da90505609247d1853f9e8c0024a5c"
server_key_iv: "32cdce2d5883a7a319301d01985be0f5"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.597846 26869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:31.610471 26869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:31.613032 26869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:31.613234 26869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:31.613843 26869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:31.613893 26869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.613919 26869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:31.613932 26869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.621878 26869 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:43945
I20260810 07:24:31.622139 27022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:43945 every 8 connection(s)
I20260810 07:24:31.622464 26869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:31.626976 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.614633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:59489 (local address 127.24.60.254:36735)
0810 07:24:31.614718 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.614721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.615035 (+ 314us) server_negotiation.cc:407] Connection header received
0810 07:24:31.615934 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.615936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.615953 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.616043 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.617272 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.618169 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.619466 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.619722 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.619793 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.623569 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.623571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.623571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.623582 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.625346 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.625836 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.625839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.625839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.625922 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.626202 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.626203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.626203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.626347 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.626427 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.626762 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.626881 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:24:31.627195 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.614623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.24.60.193:59489)
0810 07:24:31.614876 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:31.614922 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:31.615785 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:31.616156 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:31.616161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:31.616668 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:31.617108 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.617117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.618321 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.618323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.619303 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.619306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.619830 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.619831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.620024 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.620836 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:31.620850 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:31.623424 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:31.625454 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.625466 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.625469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.625745 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.625981 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.625982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.625983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.626119 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.626431 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:31.626439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:31.626608 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 07:24:31.626722 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 07:24:31.626926 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":173,"threads_started":1}
I20260810 07:24:31.627841 27023 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36735
I20260810 07:24:31.627978 27023 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:31.628191 27023 heartbeater.cc:511] Master 127.24.60.254:36735 requested a full tablet report, sending...
I20260810 07:24:31.628803 26774 ts_manager.cc:194] Registered new tserver with Master: 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945)
I20260810 07:24:31.629652 26774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:59489
I20260810 07:24:31.632525 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 26869
I20260810 07:24:31.632659 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:31.632865 24819 external_mini_cluster.cc:1428] Setting key 42f0ba7a7c230e573279d3c2ea286076
I20260810 07:24:31.636577 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.629082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:42511 (local address 127.24.60.254:36735)
0810 07:24:31.629124 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.629126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.629307 (+ 181us) server_negotiation.cc:407] Connection header received
0810 07:24:31.629391 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.629397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.629408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.629465 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.629958 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.630701 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.631574 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.631794 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.631885 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.633449 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.633451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.633451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.633460 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.635081 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.635623 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.635624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.635625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.635698 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.635977 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.635978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.635978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.636124 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.636206 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.636418 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.636497 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:24:31.636687 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.629019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.24.60.193:42511)
0810 07:24:31.629218 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:31.629231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:31.629318 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:31.629468 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:31.629469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:31.629752 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:31.629824 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.629826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.630877 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.630878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.631459 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.631461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.631884 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.631886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.631975 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.632717 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:31.632729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:31.633320 (+ 591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:31.635200 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.635202 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.635202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.635518 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.635783 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.635784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.635785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.635880 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.636220 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:31.636221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:31.636294 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 07:24:31.636381 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 07:24:31.636536 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:31.664749 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36735
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:31.753196 27033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:31.753489 27033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:31.753537 27033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:31.754802 27033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:31.754887 27033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:31.754926 27033 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 07:24:31.754957 27033 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:31.754990 27033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:31.756590 27033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:31.756937 27033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:31.757254 27033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:31.763134 27039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.763150 27040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.763167 27042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:31.763510 27033 server_base.cc:1034] running on GCE node
I20260810 07:24:31.763731 27033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:31.764005 27033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:31.765244 27033 hybrid_clock.cc:648] HybridClock initialized: now 1786346671765222 us; error 32 us; skew 500 ppm
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:31.768394 27033 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:31.768937 27033 webserver.cc:460] Webserver started at http://127.24.60.194:37533/ using document root <none> and password file <none>
I20260810 07:24:31.769280 27033 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:31.769375 27033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:31.769567 27033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:31.770814 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "4c6ecafc5f524614aabc750635cc29fb"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "b8a461d14cbc2340d9e93c74a28f8528"
server_key_iv: "699a382e10a6e0214e11fb063ab5edf3"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.771258 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6ecafc5f524614aabc750635cc29fb"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "b8a461d14cbc2340d9e93c74a28f8528"
server_key_iv: "699a382e10a6e0214e11fb063ab5edf3"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.772979 27033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:31.774084 27052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.774391 27033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:31.774523 27033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "4c6ecafc5f524614aabc750635cc29fb"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "b8a461d14cbc2340d9e93c74a28f8528"
server_key_iv: "699a382e10a6e0214e11fb063ab5edf3"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.774617 27033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:31.801929 27033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:31.804788 27033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:31.805004 27033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:31.805734 27033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:31.805806 27033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.805862 27033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:31.805949 27033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.814452 27033 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:41381
I20260810 07:24:31.814482 27180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:41381 every 8 connection(s)
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:31.814901 27033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:31.818011 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.806645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:35941 (local address 127.24.60.254:36735)
0810 07:24:31.806708 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.806711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.806951 (+ 240us) server_negotiation.cc:407] Connection header received
0810 07:24:31.807583 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.807585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.807603 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.807690 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.808696 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.809690 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.811390 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.811789 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.811903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.815083 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.815085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.815085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.815093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.816555 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.817077 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.817078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.817078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.817133 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.817444 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.817444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.817445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.817563 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.817617 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.817867 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.817944 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 07:24:31.818279 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.806608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.24.60.194:35941)
0810 07:24:31.806808 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:31.806849 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:31.807449 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:31.807768 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:31.807772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:31.808260 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:31.808569 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.808576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.809825 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.809827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.811278 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.811280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.811934 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.811935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.812095 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.812864 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:31.812875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:31.814932 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:31.816654 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.816663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.816665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.816993 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.817258 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.817259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.817259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.817383 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.817624 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:31.817630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:31.817754 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 07:24:31.817840 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:24:31.818031 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260810 07:24:31.818817 27181 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36735
I20260810 07:24:31.818914 27181 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:31.819119 27181 heartbeater.cc:511] Master 127.24.60.254:36735 requested a full tablet report, sending...
I20260810 07:24:31.819540 26776 ts_manager.cc:194] Registered new tserver with Master: 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381)
I20260810 07:24:31.819944 26776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:35941
I20260810 07:24:31.821369 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 27033
I20260810 07:24:31.821491 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:31.821671 24819 external_mini_cluster.cc:1428] Setting key 928e4bfb6696096af3c3165e88a5af02
I20260810 07:24:31.826893 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.820067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:44061 (local address 127.24.60.254:36735)
0810 07:24:31.820112 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.820114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.820233 (+ 119us) server_negotiation.cc:407] Connection header received
0810 07:24:31.820316 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.820317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.820346 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.820406 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.820727 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.821326 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.822314 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.822555 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.822611 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.823762 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.823764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.823764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.823771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.825523 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.825934 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.825935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.825936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.826019 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.826249 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.826250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.826250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.826418 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.826506 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.826704 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.826794 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:24:31.826903 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.820013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.24.60.194:44061)
0810 07:24:31.820173 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:31.820186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:31.820247 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:31.820412 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:31.820412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:31.820588 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:31.820650 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.820652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.821446 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.821447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.822222 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.822226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.822637 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.822638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.822720 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.823182 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:31.823188 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:31.823666 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:31.825610 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.825611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.825611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.825810 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.826096 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.826097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.826098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.826170 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.826496 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:31.826497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:31.826578 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 07:24:31.826662 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:24:31.826788 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:31.852959 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36735
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:31.940441 27191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:31.940659 27191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:31.940688 27191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:31.941974 27191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:31.942021 27191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:31.942042 27191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 07:24:31.942055 27191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:31.942103 27191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:31.943611 27191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:31.943876 27191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:31.944125 27191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:31.949751 27199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.949751 27197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:31.949756 27196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:31.950170 27191 server_base.cc:1034] running on GCE node
I20260810 07:24:31.950408 27191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:31.950680 27191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:31.951917 27191 hybrid_clock.cc:648] HybridClock initialized: now 1786346671951899 us; error 37 us; skew 500 ppm
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:31.954893 27191 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:31.955359 27191 webserver.cc:460] Webserver started at http://127.24.60.195:43389/ using document root <none> and password file <none>
I20260810 07:24:31.955629 27191 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:31.955680 27191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:31.955778 27191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:31.956894 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "d5488ce645ea4a028277f8193b2d05ce"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "820eaf1f4b4a6eccde7d6704433510cf"
server_key_iv: "bd2db0a5c63740e9cfa9a7509da92714"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.957295 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "d5488ce645ea4a028277f8193b2d05ce"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "820eaf1f4b4a6eccde7d6704433510cf"
server_key_iv: "bd2db0a5c63740e9cfa9a7509da92714"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.958760 27191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:31.959750 27212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.960022 27191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:31.960144 27191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "d5488ce645ea4a028277f8193b2d05ce"
format_stamp: "Formatted at 2026-08-10 07:24:31 on dist-test-slave-vxj2"
server_key: "820eaf1f4b4a6eccde7d6704433510cf"
server_key_iv: "bd2db0a5c63740e9cfa9a7509da92714"
server_key_version: "encryptionkey@0"
I20260810 07:24:31.960227 27191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:31.973358 27191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:31.975891 27191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:31.976058 27191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:31.976655 27191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:31.976694 27191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:31.976717 27191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:31.976760 27191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:24:31 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:31.985080 27191 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:44669
I20260810 07:24:31.985278 27344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:44669 every 8 connection(s)
I20260810 07:24:31.985580 27191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:31.989334 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 27191
I20260810 07:24:31.989339 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.977332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:56937 (local address 127.24.60.254:36735)
0810 07:24:31.977400 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.977402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.977675 (+ 273us) server_negotiation.cc:407] Connection header received
0810 07:24:31.978388 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.978389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.978405 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.978455 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.979561 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.980429 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.982124 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.982373 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.982483 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.985445 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.985447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.985448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.985458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.987877 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.988294 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.988294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.988295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.988354 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.988619 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.988619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.988620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.988740 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.988801 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.989054 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.989221 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:24:31.989444 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:31.989442 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.977328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.24.60.195:56937)
0810 07:24:31.977534 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:31.977575 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:31.978252 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:31.978542 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:31.978547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:31.979064 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:31.979431 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.979439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.980545 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.980547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.982014 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.982019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.982521 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.982522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.982680 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.983292 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:31.983303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:31.985286 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:31.987962 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.987969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.987971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.988215 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.988453 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.988454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.988454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.988559 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.988793 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:31.988795 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:31.988913 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 07:24:31.989021 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 07:24:31.989224 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260810 07:24:31.989657 24819 external_mini_cluster.cc:1428] Setting key a8248535616044e6f4574d2e691f3ae5
I20260810 07:24:31.990151 27345 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36735
I20260810 07:24:31.990262 27345 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:31.990495 27345 heartbeater.cc:511] Master 127.24.60.254:36735 requested a full tablet report, sending...
I20260810 07:24:31.990968 26776 ts_manager.cc:194] Registered new tserver with Master: d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:31.991528 26776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:56937
I20260810 07:24:31.997077 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.991398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:42683 (local address 127.24.60.254:36735)
0810 07:24:31.991440 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:31.991443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:31.991520 (+ 77us) server_negotiation.cc:407] Connection header received
0810 07:24:31.991591 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:31.991592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:31.991601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:31.991668 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:31.992088 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.992779 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.993601 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.993801 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.993865 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.994875 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:31.994877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:31.994877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:31.994885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:31.996103 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.996437 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.996438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.996438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.996491 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:31.996661 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:31.996661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:31.996661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:31.996762 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:31.996814 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:31.996945 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 07:24:31.997007 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 07:24:31.997143 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:31.991328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.24.60.195:42683)
0810 07:24:31.991467 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:31.991478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:31.991534 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:31.991681 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:31.991681 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:31.991900 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:31.991997 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.991999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.992871 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.992872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.993524 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:31.993526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:31.993872 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:31.993872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:31.993977 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:31.994457 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:31.994462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:31.994772 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:31.996186 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.996187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.996188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.996359 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.996552 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:31.996553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:31.996553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:31.996607 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:31.996816 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:31.996816 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:31.996856 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 07:24:31.996949 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:24:31.997037 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260810 07:24:32.002197 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:24:32 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:24:32 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:32.029742 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.021010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.24.60.254:36735)
0810 07:24:32.021077 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.021079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.021128 (+ 49us) server_negotiation.cc:407] Connection header received
0810 07:24:32.021265 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.021266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.021276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.021339 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:32.022118 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.022714 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.023610 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.023850 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.023922 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.026722 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:32.026724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:32.026724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:32.026732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:32.028392 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:32.028819 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:32.028820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:32.028820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:32.028883 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:32.029143 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:32.029144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:32.029144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:32.029349 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:32.029440 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.029591 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.029668 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:24:32.031944 26776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:32.033668 26776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:32.041132 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.036539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:41381 (local address 127.0.0.1:55194)
0810 07:24:32.036969 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.036980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.037023 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.037593 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.037594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.037596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.037667 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.037669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.038573 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.038574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.039991 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.039994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.040751 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.040845 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.040860 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.040962 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.041020 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":11,"thread_start_us":106,"threads_started":1}
I20260810 07:24:32.041339 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.037089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.24.60.195:44669)
0810 07:24:32.037410 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.037414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.037425 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:32.037466 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.037469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.037484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.037598 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.037733 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.038849 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040077 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.040874 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040978 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.041038 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.041111 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.041216 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":143,"threads_started":1}
I20260810 07:24:32.041620 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.036779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.24.60.193:43945)
0810 07:24:32.037189 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.037195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.037206 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:32.037251 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.037253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.037270 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.037350 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.037656 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.038851 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040160 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.040884 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040978 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.041038 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.041504 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.041543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":94,"threads_started":1}
I20260810 07:24:32.041702 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.036756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:44669 (local address 127.0.0.1:33006)
0810 07:24:32.037117 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.037129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.037217 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.037596 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.037596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.037598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.037668 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.037669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.038968 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.038969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.039967 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.039970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.040976 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.041032 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.041035 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.041096 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.041127 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":28,"thread_start_us":204,"threads_started":1}
I20260810 07:24:32.041911 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.036516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43945 (local address 127.0.0.1:48032)
0810 07:24:32.036704 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.036753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.036838 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.037457 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.037461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.037463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.037548 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.037551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.038967 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.038968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.040095 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.040096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.041212 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.041212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.041271 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.041275 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.041340 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.041382 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":26,"thread_start_us":75,"threads_started":1}
I20260810 07:24:32.042423 27270 tablet_service.cc:1467] Processing CreateTablet for tablet 077605aff6914a789ef0e73ad31d94bc (DEFAULT_TABLE table=test-table [id=1fd58d31fca640068962b5dbdab4655a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:32.042732 27270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077605aff6914a789ef0e73ad31d94bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.044977 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.036671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.24.60.194:41381)
0810 07:24:32.037367 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.037372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.037383 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:32.037424 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.037426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.037442 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.037517 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.037768 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.038474 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040093 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.040653 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.040751 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.040817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.044768 (+ 3951us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.044848 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":97,"threads_started":1}
I20260810 07:24:32.045697 27371 tablet_bootstrap.cc:492] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Bootstrap starting.
I20260810 07:24:32.045737 26944 tablet_service.cc:1467] Processing CreateTablet for tablet 077605aff6914a789ef0e73ad31d94bc (DEFAULT_TABLE table=test-table [id=1fd58d31fca640068962b5dbdab4655a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:32.045744 27107 tablet_service.cc:1467] Processing CreateTablet for tablet 077605aff6914a789ef0e73ad31d94bc (DEFAULT_TABLE table=test-table [id=1fd58d31fca640068962b5dbdab4655a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:32.046118 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077605aff6914a789ef0e73ad31d94bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.046157 27107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077605aff6914a789ef0e73ad31d94bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.046830 27371 tablet_bootstrap.cc:654] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.047186 27371 log.cc:824] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:32.048101 27371 tablet_bootstrap.cc:492] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: No bootstrap required, opened a new log
I20260810 07:24:32.048168 27371 ts_tablet_manager.cc:1397] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:24:32.049337 27375 tablet_bootstrap.cc:492] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Bootstrap starting.
I20260810 07:24:32.049944 27375 tablet_bootstrap.cc:654] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.050349 27375 log.cc:824] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:32.050364 27376 tablet_bootstrap.cc:492] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Bootstrap starting.
I20260810 07:24:32.050433 27371 raft_consensus.cc:348] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.050678 27371 raft_consensus.cc:374] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.050726 27371 raft_consensus.cc:729] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5488ce645ea4a028277f8193b2d05ce, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.050904 27371 consensus_queue.cc:260] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.051105 27376 tablet_bootstrap.cc:654] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.051200 27375 tablet_bootstrap.cc:492] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: No bootstrap required, opened a new log
I20260810 07:24:32.051220 27371 ts_tablet_manager.cc:1428] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:32.051285 27375 ts_tablet_manager.cc:1397] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:32.051407 27345 heartbeater.cc:503] Master 127.24.60.254:36735 was elected leader, sending a full tablet report...
I20260810 07:24:32.051494 27376 log.cc:824] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:32.052323 27376 tablet_bootstrap.cc:492] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: No bootstrap required, opened a new log
I20260810 07:24:32.052423 27376 ts_tablet_manager.cc:1397] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:32.053442 27375 raft_consensus.cc:348] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.053586 27375 raft_consensus.cc:374] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.053608 27375 raft_consensus.cc:729] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.053736 27375 consensus_queue.cc:260] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.054056 27375 ts_tablet_manager.cc:1428] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:32.054095 27181 heartbeater.cc:503] Master 127.24.60.254:36735 was elected leader, sending a full tablet report...
I20260810 07:24:32.054594 27376 raft_consensus.cc:348] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.054756 27376 raft_consensus.cc:374] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.054795 27376 raft_consensus.cc:729] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac5d35a259f4961950c3d1b85c35582, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.054942 27376 consensus_queue.cc:260] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.055258 27376 ts_tablet_manager.cc:1428] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:32.055330 27023 heartbeater.cc:503] Master 127.24.60.254:36735 was elected leader, sending a full tablet report...
W20260810 07:24:32.065531 27182 tablet.cc:2369] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:32.112345 27380 raft_consensus.cc:482] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:32.112490 27380 raft_consensus.cc:504] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.112957 27380 leader_election.cc:288] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:32.117695 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.113592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:33947 (local address 127.24.60.195:44669)
0810 07:24:32.113665 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.113667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.113782 (+ 115us) server_negotiation.cc:407] Connection header received
0810 07:24:32.113840 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.113841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.113849 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.113895 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.114064 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.114843 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.116555 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.117180 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.117269 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.117282 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.117568 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.117619 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:24:32.117695 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.113334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:40071 (local address 127.24.60.193:43945)
0810 07:24:32.113431 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.113433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.113469 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:24:32.113575 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.113577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.113586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.113658 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.114074 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.114842 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.116555 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.117246 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.117315 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.117324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.117568 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.117617 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 07:24:32.117697 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.113319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43945 (local address 127.24.60.194:40071)
0810 07:24:32.113400 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.113416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.113492 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.113713 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.113715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.113718 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.113870 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.113873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.114961 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.114962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.116416 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.116420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.117335 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.117337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.117446 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.117451 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.117523 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.117581 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":68}
I20260810 07:24:32.117708 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.113494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:44669 (local address 127.24.60.194:33947)
0810 07:24:32.113692 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.113704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.113772 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.113909 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.113910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.113911 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.113977 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.113985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.114961 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.114962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.116416 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.116420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.117335 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.117337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.117446 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.117451 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.117524 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.117581 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":85,"thread_start_us":143,"threads_started":1}
I20260810 07:24:32.118299 26970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077605aff6914a789ef0e73ad31d94bc" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582" is_pre_election: true
I20260810 07:24:32.118454 26970 raft_consensus.cc:2454] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 0.
I20260810 07:24:32.118702 27053 leader_election.cc:302] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, 8ac5d35a259f4961950c3d1b85c35582; no voters:
I20260810 07:24:32.118886 27380 raft_consensus.cc:2781] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:32.118865 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077605aff6914a789ef0e73ad31d94bc" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce" is_pre_election: true
I20260810 07:24:32.118975 27380 raft_consensus.cc:482] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:32.119002 27380 raft_consensus.cc:3037] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.119041 27295 raft_consensus.cc:2454] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 0.
I20260810 07:24:32.120142 27380 raft_consensus.cc:504] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.120260 27380 leader_election.cc:288] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 election: Requested vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:32.120523 26970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077605aff6914a789ef0e73ad31d94bc" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582"
I20260810 07:24:32.120568 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077605aff6914a789ef0e73ad31d94bc" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce"
I20260810 07:24:32.120692 27295 raft_consensus.cc:3037] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.120682 26970 raft_consensus.cc:3037] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.121523 27295 raft_consensus.cc:2454] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 1.
I20260810 07:24:32.121714 27053 leader_election.cc:302] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, d5488ce645ea4a028277f8193b2d05ce; no voters:
I20260810 07:24:32.121851 26970 raft_consensus.cc:2454] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 1.
I20260810 07:24:32.121876 27380 raft_consensus.cc:2781] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:32.122117 27380 raft_consensus.cc:686] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 1 LEADER]: Becoming Leader. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Running, Role: LEADER
I20260810 07:24:32.122236 27380 consensus_queue.cc:237] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
W20260810 07:24:32.123463 27024 tablet.cc:2369] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:32.123303 26776 catalog_manager.cc:5495] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194). New cstate: current_term: 1 leader_uuid: "4c6ecafc5f524614aabc750635cc29fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } health_report { overall_health: UNKNOWN } } }
Aug 10 07:24:32 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:32.162119 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.154206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.24.60.254:36735)
0810 07:24:32.154309 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.154312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.154451 (+ 139us) server_negotiation.cc:407] Connection header received
0810 07:24:32.154517 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.154518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.154529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.154609 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:32.155139 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.155686 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.156401 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.156589 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.156681 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.158996 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:32.158998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:32.158998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:32.159006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:32.160629 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:32.161069 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:32.161070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:32.161071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:32.161138 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:32.161373 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:32.161374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:32.161375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:32.161561 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:32.161637 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.161816 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.161996 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":70}
I20260810 07:24:32.162197 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.154101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36735 (local address 127.0.0.1:33668)
0810 07:24:32.154342 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.154354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.154458 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.154614 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.154615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.154980 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:32.155036 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.155038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.155774 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.155776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.156312 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.156313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.156694 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.156695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.156757 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.157384 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:32.157389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:32.158857 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:32.160755 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:32.160767 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:32.160770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:32.160986 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:32.161240 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:32.161241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:32.161241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:32.161303 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:32.161627 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:32.161637 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:32.161710 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.161785 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.162067 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":144,"thread_start_us":121,"threads_started":1}
I20260810 07:24:32.164680 26776 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:33668:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 07:24:32.164878 26776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:32.166522 27107 tablet_service.cc:1467] Processing CreateTablet for tablet ad39e999c8cd411aa3ba94c8d0c1a795 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=deee82d658a345988f2d0b21d4821db2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:32.166504 26944 tablet_service.cc:1467] Processing CreateTablet for tablet ad39e999c8cd411aa3ba94c8d0c1a795 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=deee82d658a345988f2d0b21d4821db2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:32.166608 27270 tablet_service.cc:1467] Processing CreateTablet for tablet ad39e999c8cd411aa3ba94c8d0c1a795 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=deee82d658a345988f2d0b21d4821db2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:32.166723 27107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad39e999c8cd411aa3ba94c8d0c1a795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.166723 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad39e999c8cd411aa3ba94c8d0c1a795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.166793 27270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad39e999c8cd411aa3ba94c8d0c1a795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.168205 27371 tablet_bootstrap.cc:492] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: Bootstrap starting.
I20260810 07:24:32.168331 27376 tablet_bootstrap.cc:492] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: Bootstrap starting.
I20260810 07:24:32.168792 27376 tablet_bootstrap.cc:654] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.168862 27371 tablet_bootstrap.cc:654] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.169126 27375 tablet_bootstrap.cc:492] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: Bootstrap starting.
I20260810 07:24:32.169572 27371 tablet_bootstrap.cc:492] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: No bootstrap required, opened a new log
I20260810 07:24:32.169642 27371 ts_tablet_manager.cc:1397] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:32.169632 27375 tablet_bootstrap.cc:654] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.169809 27371 raft_consensus.cc:348] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.169899 27371 raft_consensus.cc:374] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.169924 27371 raft_consensus.cc:729] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5488ce645ea4a028277f8193b2d05ce, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.169977 27371 consensus_queue.cc:260] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.170074 27371 ts_tablet_manager.cc:1428] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 07:24:32.170221 27378 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.170287 27376 tablet_bootstrap.cc:492] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: No bootstrap required, opened a new log
I20260810 07:24:32.170312 27378 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:32.170331 27376 ts_tablet_manager.cc:1397] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:32.170517 27376 raft_consensus.cc:348] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.170604 27376 raft_consensus.cc:374] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.170624 27376 raft_consensus.cc:729] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac5d35a259f4961950c3d1b85c35582, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.170663 27376 consensus_queue.cc:260] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.170771 27376 ts_tablet_manager.cc:1428] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:32.170810 27381 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.170941 27381 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:32.171186 27375 tablet_bootstrap.cc:492] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: No bootstrap required, opened a new log
I20260810 07:24:32.171232 27375 ts_tablet_manager.cc:1397] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:32.171417 27375 raft_consensus.cc:348] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.171504 27375 raft_consensus.cc:374] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.171528 27375 raft_consensus.cc:729] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.171584 27375 consensus_queue.cc:260] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.171721 27375 ts_tablet_manager.cc:1428] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:32.171774 27380 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.171900 27380 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 07:24:32.237355 27346 tablet.cc:2369] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:32.417004 27378 raft_consensus.cc:482] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:32.417147 27378 raft_consensus.cc:504] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.417639 27378 leader_election.cc:288] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381)
I20260810 07:24:32.421242 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.418018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:37029 (local address 127.24.60.194:41381)
0810 07:24:32.418069 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.418072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.418293 (+ 221us) server_negotiation.cc:407] Connection header received
0810 07:24:32.418366 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.418367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.418376 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.418453 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.418661 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.419268 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.420424 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.420794 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.420890 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.420901 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.421097 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.421140 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:32.421285 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.417892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43945 (local address 127.24.60.195:50105)
0810 07:24:32.417973 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.417989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.418068 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.418267 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.418268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.418269 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.418365 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.418367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.419207 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.419208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.420247 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.420250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.420930 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.420932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.421038 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.421042 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.421116 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.421183 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 07:24:32.421277 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.418042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:50105 (local address 127.24.60.193:43945)
0810 07:24:32.418109 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.418111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.418138 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:24:32.418174 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.418175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.418189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.418261 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:32.418451 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.419114 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.420349 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.420813 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.420927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.420936 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.421127 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.421198 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:24:32.421319 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.417907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:41381 (local address 127.24.60.195:37029)
0810 07:24:32.418193 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.418202 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.418309 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.418455 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.418456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.418458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:32.418580 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.418586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.419379 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.419380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.420346 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.420349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.420917 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.420919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.421011 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.421015 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.421089 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.421133 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":92,"thread_start_us":135,"threads_started":1}
I20260810 07:24:32.421556 26970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad39e999c8cd411aa3ba94c8d0c1a795" candidate_uuid: "d5488ce645ea4a028277f8193b2d05ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582" is_pre_election: true
I20260810 07:24:32.421677 26970 raft_consensus.cc:2454] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5488ce645ea4a028277f8193b2d05ce in term 0.
I20260810 07:24:32.421649 27133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad39e999c8cd411aa3ba94c8d0c1a795" candidate_uuid: "d5488ce645ea4a028277f8193b2d05ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6ecafc5f524614aabc750635cc29fb" is_pre_election: true
I20260810 07:24:32.421770 27133 raft_consensus.cc:2454] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5488ce645ea4a028277f8193b2d05ce in term 0.
I20260810 07:24:32.421924 27213 leader_election.cc:302] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac5d35a259f4961950c3d1b85c35582, d5488ce645ea4a028277f8193b2d05ce; no voters:
I20260810 07:24:32.422082 27378 raft_consensus.cc:2781] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:32.422125 27378 raft_consensus.cc:482] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:32.422140 27378 raft_consensus.cc:3037] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.422848 27378 raft_consensus.cc:504] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.423004 27378 leader_election.cc:288] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [CANDIDATE]: Term 1 election: Requested vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381)
I20260810 07:24:32.423193 26970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad39e999c8cd411aa3ba94c8d0c1a795" candidate_uuid: "d5488ce645ea4a028277f8193b2d05ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582"
I20260810 07:24:32.423185 27133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad39e999c8cd411aa3ba94c8d0c1a795" candidate_uuid: "d5488ce645ea4a028277f8193b2d05ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6ecafc5f524614aabc750635cc29fb"
I20260810 07:24:32.423271 26970 raft_consensus.cc:3037] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.423292 27133 raft_consensus.cc:3037] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.423919 26970 raft_consensus.cc:2454] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5488ce645ea4a028277f8193b2d05ce in term 1.
I20260810 07:24:32.424098 27213 leader_election.cc:302] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac5d35a259f4961950c3d1b85c35582, d5488ce645ea4a028277f8193b2d05ce; no voters:
I20260810 07:24:32.424213 27133 raft_consensus.cc:2454] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5488ce645ea4a028277f8193b2d05ce in term 1.
I20260810 07:24:32.424230 27378 raft_consensus.cc:2781] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:32.424454 27378 raft_consensus.cc:686] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [term 1 LEADER]: Becoming Leader. State: Replica: d5488ce645ea4a028277f8193b2d05ce, State: Running, Role: LEADER
I20260810 07:24:32.424592 27378 consensus_queue.cc:237] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.425179 27399 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: TxnStatusTablet state changed. Reason: New leader d5488ce645ea4a028277f8193b2d05ce. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.425279 27399 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:32.425565 27401 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 07:24:32.425699 27401 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 07:24:32.425639 26774 catalog_manager.cc:5495] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce reported cstate change: term changed from 0 to 1, leader changed from <none> to d5488ce645ea4a028277f8193b2d05ce (127.24.60.195). New cstate: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } health_report { overall_health: HEALTHY } } }
I20260810 07:24:32.459378 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.456063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.24.60.195:44669)
0810 07:24:32.456140 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.456142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.456153 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:32.456197 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.456198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.456220 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.456331 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:32.456507 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.457302 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.458231 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.458547 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.458624 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.458886 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.459077 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.459206 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.459260 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.459285 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:24:32.459373 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.455751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:44669 (local address 127.0.0.1:33012)
0810 07:24:32.455879 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.455892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.455956 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.456358 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.456360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.456362 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:32.456418 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.456420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.457420 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.457421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.458127 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.458130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.458650 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.458652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.458718 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.458738 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.459192 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.459204 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.459253 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.459304 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 07:24:32.461627 27133 raft_consensus.cc:1261] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Refusing update from remote peer d5488ce645ea4a028277f8193b2d05ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:32.461625 26970 raft_consensus.cc:1261] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Refusing update from remote peer d5488ce645ea4a028277f8193b2d05ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:32.461987 27399 consensus_queue.cc:1035] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:32.462118 27378 consensus_queue.cc:1035] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:32.463631 27380 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: TxnStatusTablet state changed. Reason: New leader d5488ce645ea4a028277f8193b2d05ce. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.463729 27380 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:32.464057 27381 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: TxnStatusTablet state changed. Reason: New leader d5488ce645ea4a028277f8193b2d05ce. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.464160 27381 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:32.465106 27378 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.465247 27378 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:32.465330 27381 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.465425 27381 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P 8ac5d35a259f4961950c3d1b85c35582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:32.465487 27399 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.465559 27399 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P d5488ce645ea4a028277f8193b2d05ce: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:32.465416 27407 mvcc.cc:205] Tried to move back new op lower bound from 7316875970399313920 to 7316875970251890688. Current Snapshot: MvccSnapshot[applied={T|T < 7316875970399313920}]
I20260810 07:24:32.465775 27404 mvcc.cc:205] Tried to move back new op lower bound from 7316875970399313920 to 7316875970251890688. Current Snapshot: MvccSnapshot[applied={T|T < 7316875970399313920}]
I20260810 07:24:32.466048 27380 tablet_replica.cc:439] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5488ce645ea4a028277f8193b2d05ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } } }
I20260810 07:24:32.466140 27380 tablet_replica.cc:442] T ad39e999c8cd411aa3ba94c8d0c1a795 P 4c6ecafc5f524614aabc750635cc29fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:32.467190 27405 mvcc.cc:205] Tried to move back new op lower bound from 7316875970399313920 to 7316875970251890688. Current Snapshot: MvccSnapshot[applied={T|T < 7316875970399313920 or (T in {7316875970399313920})}]
I20260810 07:24:32.478595 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.475856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.24.60.194:41381)
0810 07:24:32.475922 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.475924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.475933 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:32.475983 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.475984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.476004 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.476084 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:32.476253 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.476780 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.477769 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.477969 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.478045 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.478228 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.478337 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.478439 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.478464 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.478495 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:24:32.484413 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.481559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:43449 (local address 127.24.60.195:44669)
0810 07:24:32.481624 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.481626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.481687 (+ 61us) server_negotiation.cc:407] Connection header received
0810 07:24:32.481741 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.481742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.481754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.481817 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:32.481913 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.482644 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.483460 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.483751 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.483839 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.484088 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.484126 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.484183 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.484306 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.484328 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:24:32.484428 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.481475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:44669 (local address 127.24.60.194:43449)
0810 07:24:32.481634 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.481644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.481696 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.481821 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.481821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.481823 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:32.481875 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.481876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.482747 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.482748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.483367 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.483369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.483866 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.483868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.483934 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.483958 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.484222 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.484225 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.484285 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.484339 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260810 07:24:32.487380 26970 raft_consensus.cc:1261] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Refusing update from remote peer 4c6ecafc5f524614aabc750635cc29fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:32.487489 27295 raft_consensus.cc:1261] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Refusing update from remote peer 4c6ecafc5f524614aabc750635cc29fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:32.487653 27380 consensus_queue.cc:1035] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:32.487810 27383 consensus_queue.cc:1035] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:32.497694 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.494749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:41381 (local address 127.24.60.195:46761)
0810 07:24:32.495009 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:32.495023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:32.495091 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:32.495232 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:32.495233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:32.495234 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:32.495287 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.495289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.495984 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.495985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.496679 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:32.496682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.497259 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.497260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:32.497333 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.497355 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.497553 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.497555 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 07:24:32.497590 (+ 35us) client_negotiation.cc:240] Negotiation successful
0810 07:24:32.497632 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260810 07:24:32.497696 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:32.494978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:46761 (local address 127.24.60.194:41381)
0810 07:24:32.495028 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:32.495030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:32.495068 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:24:32.495152 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:32.495153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:32.495168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:32.495229 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:32.495344 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.495882 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.496836 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:32.497185 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:32.497262 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:32.497453 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:32.497489 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:32.497553 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:32.497593 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 07:24:32.497619 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 07:24:32.513048 26776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:32.513217 26776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:32.514186 26776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Sending DeleteTablet for 3 replicas of tablet 077605aff6914a789ef0e73ad31d94bc
I20260810 07:24:32.514794 27270 tablet_service.cc:1514] Processing DeleteTablet for tablet 077605aff6914a789ef0e73ad31d94bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:32 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:33006
I20260810 07:24:32.514824 27107 tablet_service.cc:1514] Processing DeleteTablet for tablet 077605aff6914a789ef0e73ad31d94bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:32 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:55194
I20260810 07:24:32.515326 26944 tablet_service.cc:1514] Processing DeleteTablet for tablet 077605aff6914a789ef0e73ad31d94bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:32 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:48032
I20260810 07:24:32.515635 27431 tablet_replica.cc:330] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: stopping tablet replica
I20260810 07:24:32.515609 26776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 07:24:32.515779 27431 raft_consensus.cc:2229] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 07:24:32.515801 26776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:32.515852 27431 raft_consensus.cc:2258] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:32.516538 27431 ts_tablet_manager.cc:1905] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:32.517405 27107 tablet_service.cc:1467] Processing CreateTablet for tablet d3d4d7dbedef418b8092e62a55f0eb18 (DEFAULT_TABLE table=test-table [id=efc808cda9314947a9d59db2ce87ace8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:32.517565 27107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4d7dbedef418b8092e62a55f0eb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.519152 27375 tablet_bootstrap.cc:492] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Bootstrap starting.
I20260810 07:24:32.519325 27270 tablet_service.cc:1467] Processing CreateTablet for tablet d3d4d7dbedef418b8092e62a55f0eb18 (DEFAULT_TABLE table=test-table [id=efc808cda9314947a9d59db2ce87ace8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:32.519485 27270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4d7dbedef418b8092e62a55f0eb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.519649 27427 tablet_replica.cc:330] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: stopping tablet replica
I20260810 07:24:32.519693 27375 tablet_bootstrap.cc:654] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.519799 27427 raft_consensus.cc:2229] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:32.519937 27427 raft_consensus.cc:2258] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:32.521059 27371 tablet_bootstrap.cc:492] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Bootstrap starting.
I20260810 07:24:32.521435 26944 tablet_service.cc:1467] Processing CreateTablet for tablet d3d4d7dbedef418b8092e62a55f0eb18 (DEFAULT_TABLE table=test-table [id=efc808cda9314947a9d59db2ce87ace8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:32.521584 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4d7dbedef418b8092e62a55f0eb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:32.521692 27371 tablet_bootstrap.cc:654] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.521682 27427 ts_tablet_manager.cc:1905] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:32.523099 27376 tablet_bootstrap.cc:492] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Bootstrap starting.
I20260810 07:24:32.523546 27376 tablet_bootstrap.cc:654] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:32.524185 27376 tablet_bootstrap.cc:492] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: No bootstrap required, opened a new log
I20260810 07:24:32.524245 27376 ts_tablet_manager.cc:1397] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:32.524430 27376 raft_consensus.cc:348] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.524515 27376 raft_consensus.cc:374] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.524538 27431 ts_tablet_manager.cc:1918] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:32.524636 27431 log.cc:1196] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/077605aff6914a789ef0e73ad31d94bc
I20260810 07:24:32.524538 27376 raft_consensus.cc:729] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac5d35a259f4961950c3d1b85c35582, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.524952 27431 ts_tablet_manager.cc:1939] T 077605aff6914a789ef0e73ad31d94bc P 8ac5d35a259f4961950c3d1b85c35582: Deleting consensus metadata
I20260810 07:24:32.524935 27376 consensus_queue.cc:260] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.525043 27376 ts_tablet_manager.cc:1428] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:32.528008 27427 ts_tablet_manager.cc:1918] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:32.528158 27427 log.cc:1196] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/077605aff6914a789ef0e73ad31d94bc
I20260810 07:24:32.528446 27427 ts_tablet_manager.cc:1939] T 077605aff6914a789ef0e73ad31d94bc P 4c6ecafc5f524614aabc750635cc29fb: Deleting consensus metadata
I20260810 07:24:32.528936 26766 catalog_manager.cc:4841] TS 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381): tablet 077605aff6914a789ef0e73ad31d94bc (table test-table [id=1fd58d31fca640068962b5dbdab4655a]) successfully deleted
I20260810 07:24:32.529297 26765 catalog_manager.cc:4841] TS 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945): tablet 077605aff6914a789ef0e73ad31d94bc (table test-table [id=1fd58d31fca640068962b5dbdab4655a]) successfully deleted
I20260810 07:24:32.529716 27375 tablet_bootstrap.cc:492] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: No bootstrap required, opened a new log
I20260810 07:24:32.529772 27375 ts_tablet_manager.cc:1397] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260810 07:24:32.529932 27375 raft_consensus.cc:348] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.530009 27375 raft_consensus.cc:374] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.530030 27375 raft_consensus.cc:729] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.530083 27375 consensus_queue.cc:260] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.530175 27375 ts_tablet_manager.cc:1428] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:32.530560 27430 tablet_replica.cc:330] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: stopping tablet replica
I20260810 07:24:32.530740 27430 raft_consensus.cc:2229] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:32.530862 27430 raft_consensus.cc:2258] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:32.531538 27430 ts_tablet_manager.cc:1905] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:32.532943 27371 tablet_bootstrap.cc:492] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: No bootstrap required, opened a new log
I20260810 07:24:32.533000 27371 ts_tablet_manager.cc:1397] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260810 07:24:32.533177 27371 raft_consensus.cc:348] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.533282 27371 raft_consensus.cc:374] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:32.533321 27371 raft_consensus.cc:729] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5488ce645ea4a028277f8193b2d05ce, State: Initialized, Role: FOLLOWER
I20260810 07:24:32.533383 27371 consensus_queue.cc:260] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.533491 27371 ts_tablet_manager.cc:1428] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:32.534912 27430 ts_tablet_manager.cc:1918] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:32.534999 27430 log.cc:1196] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/077605aff6914a789ef0e73ad31d94bc
I20260810 07:24:32.535403 27430 ts_tablet_manager.cc:1939] T 077605aff6914a789ef0e73ad31d94bc P d5488ce645ea4a028277f8193b2d05ce: Deleting consensus metadata
I20260810 07:24:32.535970 26765 catalog_manager.cc:4841] TS d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669): tablet 077605aff6914a789ef0e73ad31d94bc (table test-table [id=1fd58d31fca640068962b5dbdab4655a]) successfully deleted
I20260810 07:24:32.659811 27380 raft_consensus.cc:482] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:32.659955 27380 raft_consensus.cc:504] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.660169 27380 leader_election.cc:288] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:32.660557 26968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4d7dbedef418b8092e62a55f0eb18" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582" is_pre_election: true
I20260810 07:24:32.660586 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4d7dbedef418b8092e62a55f0eb18" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce" is_pre_election: true
I20260810 07:24:32.660696 26968 raft_consensus.cc:2454] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 0.
I20260810 07:24:32.660691 27295 raft_consensus.cc:2454] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 0.
I20260810 07:24:32.660900 27053 leader_election.cc:302] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, 8ac5d35a259f4961950c3d1b85c35582; no voters:
I20260810 07:24:32.661037 27380 raft_consensus.cc:2781] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:32.661094 27380 raft_consensus.cc:482] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:32.661109 27380 raft_consensus.cc:3037] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.661906 27380 raft_consensus.cc:504] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.662019 27380 leader_election.cc:288] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 election: Requested vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:32.662240 26968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4d7dbedef418b8092e62a55f0eb18" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582"
I20260810 07:24:32.662281 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4d7dbedef418b8092e62a55f0eb18" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce"
I20260810 07:24:32.662346 26968 raft_consensus.cc:3037] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.662354 27295 raft_consensus.cc:3037] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:32.663273 26968 raft_consensus.cc:2454] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 1.
I20260810 07:24:32.663273 27295 raft_consensus.cc:2454] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 1.
I20260810 07:24:32.663458 27053 leader_election.cc:302] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, 8ac5d35a259f4961950c3d1b85c35582; no voters:
I20260810 07:24:32.663583 27380 raft_consensus.cc:2781] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:32.663657 27380 raft_consensus.cc:686] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 1 LEADER]: Becoming Leader. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Running, Role: LEADER
I20260810 07:24:32.663709 27380 consensus_queue.cc:237] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:32.664439 26776 catalog_manager.cc:5495] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194). New cstate: current_term: 1 leader_uuid: "4c6ecafc5f524614aabc750635cc29fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:32.707558 27295 raft_consensus.cc:1261] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Refusing update from remote peer 4c6ecafc5f524614aabc750635cc29fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:32.707558 26968 raft_consensus.cc:1261] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Refusing update from remote peer 4c6ecafc5f524614aabc750635cc29fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:32.707818 27380 consensus_queue.cc:1035] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:32.707988 27420 consensus_queue.cc:1035] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:24:32.712975 27405 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 07:24:34.233029 26773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:34.233209 26773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:34.234525 26773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Sending DeleteTablet for 3 replicas of tablet d3d4d7dbedef418b8092e62a55f0eb18
I20260810 07:24:34.234887 27107 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d4d7dbedef418b8092e62a55f0eb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:34 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:55194
I20260810 07:24:34.234972 27270 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d4d7dbedef418b8092e62a55f0eb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:34 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:33006
I20260810 07:24:34.234961 26944 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d4d7dbedef418b8092e62a55f0eb18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:34 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:48032
I20260810 07:24:34.235162 27462 tablet_replica.cc:330] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: stopping tablet replica
I20260810 07:24:34.235220 27463 tablet_replica.cc:330] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: stopping tablet replica
I20260810 07:24:34.235261 27462 raft_consensus.cc:2229] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:34.235294 27463 raft_consensus.cc:2229] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:34.235369 27462 raft_consensus.cc:2258] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:34.235437 27464 tablet_replica.cc:330] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: stopping tablet replica
I20260810 07:24:34.235505 27464 raft_consensus.cc:2229] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:34.235551 27464 raft_consensus.cc:2258] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:34.235632 27463 raft_consensus.cc:2258] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:34.235656 27462 ts_tablet_manager.cc:1905] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:34.235934 27463 ts_tablet_manager.cc:1905] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:34.237643 27463 ts_tablet_manager.cc:1918] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:24:34.237792 27463 log.cc:1196] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/d3d4d7dbedef418b8092e62a55f0eb18
I20260810 07:24:34.238059 27462 ts_tablet_manager.cc:1918] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:24:34.238104 27463 ts_tablet_manager.cc:1939] T d3d4d7dbedef418b8092e62a55f0eb18 P d5488ce645ea4a028277f8193b2d05ce: Deleting consensus metadata
I20260810 07:24:34.238207 27462 log.cc:1196] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/d3d4d7dbedef418b8092e62a55f0eb18
I20260810 07:24:34.238570 27462 ts_tablet_manager.cc:1939] T d3d4d7dbedef418b8092e62a55f0eb18 P 4c6ecafc5f524614aabc750635cc29fb: Deleting consensus metadata
I20260810 07:24:34.238590 26765 catalog_manager.cc:4841] TS d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669): tablet d3d4d7dbedef418b8092e62a55f0eb18 (table test-table [id=efc808cda9314947a9d59db2ce87ace8]) successfully deleted
I20260810 07:24:34.239030 26766 catalog_manager.cc:4841] TS 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381): tablet d3d4d7dbedef418b8092e62a55f0eb18 (table test-table [id=efc808cda9314947a9d59db2ce87ace8]) successfully deleted
I20260810 07:24:34.239187 27464 ts_tablet_manager.cc:1905] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:34.240813 27464 ts_tablet_manager.cc:1918] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:24:34.240878 27464 log.cc:1196] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/d3d4d7dbedef418b8092e62a55f0eb18
I20260810 07:24:34.241125 27464 ts_tablet_manager.cc:1939] T d3d4d7dbedef418b8092e62a55f0eb18 P 8ac5d35a259f4961950c3d1b85c35582: Deleting consensus metadata
I20260810 07:24:34.241757 26765 catalog_manager.cc:4841] TS 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945): tablet d3d4d7dbedef418b8092e62a55f0eb18 (table test-table [id=efc808cda9314947a9d59db2ce87ace8]) successfully deleted
Aug 10 07:24:34 dist-test-slave-vxj2 krb5kdc[26734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:24:34 dist-test-slave-vxj2 krb5kdc[26734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:34.263422 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.254234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.24.60.254:36735)
0810 07:24:34.254404 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.254406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.254461 (+ 55us) server_negotiation.cc:407] Connection header received
0810 07:24:34.254573 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.254575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.254592 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.254677 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:34.255313 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.255948 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.256784 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.257072 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.257143 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.259903 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:34.259904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:34.259904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:34.259921 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:34.261838 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:34.262296 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:34.262297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:34.262298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:34.262372 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:34.262643 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:34.262644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:34.262645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:34.262916 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:34.263037 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.263195 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.263299 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260810 07:24:34.265460 26773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:34.265669 26773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:34.267381 27107 tablet_service.cc:1467] Processing CreateTablet for tablet 563494cf6c034f3fb35176691e8da9f6 (DEFAULT_TABLE table=test-table [id=1a0aaf3f7063415eb7b78611cf66fe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:34.267385 26944 tablet_service.cc:1467] Processing CreateTablet for tablet 563494cf6c034f3fb35176691e8da9f6 (DEFAULT_TABLE table=test-table [id=1a0aaf3f7063415eb7b78611cf66fe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:34.267419 27270 tablet_service.cc:1467] Processing CreateTablet for tablet 563494cf6c034f3fb35176691e8da9f6 (DEFAULT_TABLE table=test-table [id=1a0aaf3f7063415eb7b78611cf66fe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:34.267566 27270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563494cf6c034f3fb35176691e8da9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:34.267566 27107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563494cf6c034f3fb35176691e8da9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:34.267580 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563494cf6c034f3fb35176691e8da9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:34.268810 27476 tablet_bootstrap.cc:492] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Bootstrap starting.
I20260810 07:24:34.269040 27477 tablet_bootstrap.cc:492] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Bootstrap starting.
I20260810 07:24:34.269237 27478 tablet_bootstrap.cc:492] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Bootstrap starting.
I20260810 07:24:34.269330 27476 tablet_bootstrap.cc:654] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:34.269477 27477 tablet_bootstrap.cc:654] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:34.269670 27478 tablet_bootstrap.cc:654] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:34.270073 27476 tablet_bootstrap.cc:492] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: No bootstrap required, opened a new log
I20260810 07:24:34.270149 27476 ts_tablet_manager.cc:1397] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:34.270339 27476 raft_consensus.cc:348] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.270434 27476 raft_consensus.cc:374] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:34.270471 27476 raft_consensus.cc:729] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac5d35a259f4961950c3d1b85c35582, State: Initialized, Role: FOLLOWER
I20260810 07:24:34.270533 27477 tablet_bootstrap.cc:492] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: No bootstrap required, opened a new log
I20260810 07:24:34.270524 27476 consensus_queue.cc:260] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.270584 27477 ts_tablet_manager.cc:1397] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:34.270721 27476 ts_tablet_manager.cc:1428] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:34.270753 27478 tablet_bootstrap.cc:492] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: No bootstrap required, opened a new log
I20260810 07:24:34.270738 27477 raft_consensus.cc:348] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.270807 27478 ts_tablet_manager.cc:1397] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:34.270808 27477 raft_consensus.cc:374] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:34.270834 27477 raft_consensus.cc:729] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5488ce645ea4a028277f8193b2d05ce, State: Initialized, Role: FOLLOWER
I20260810 07:24:34.270895 27477 consensus_queue.cc:260] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.270934 27478 raft_consensus.cc:348] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.271005 27477 ts_tablet_manager.cc:1428] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:34.271020 27478 raft_consensus.cc:374] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:34.271044 27478 raft_consensus.cc:729] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Initialized, Role: FOLLOWER
I20260810 07:24:34.271093 27478 consensus_queue.cc:260] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.271225 27478 ts_tablet_manager.cc:1428] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:34.341770 27453 raft_consensus.cc:482] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:34.341960 27453 raft_consensus.cc:504] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.342154 27453 leader_election.cc:288] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:34.342473 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563494cf6c034f3fb35176691e8da9f6" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce" is_pre_election: true
I20260810 07:24:34.342576 27295 raft_consensus.cc:2454] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 0.
I20260810 07:24:34.342511 26968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563494cf6c034f3fb35176691e8da9f6" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582" is_pre_election: true
I20260810 07:24:34.342651 26968 raft_consensus.cc:2454] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 0.
I20260810 07:24:34.342788 27053 leader_election.cc:302] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, d5488ce645ea4a028277f8193b2d05ce; no voters:
I20260810 07:24:34.342921 27453 raft_consensus.cc:2781] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:34.342964 27453 raft_consensus.cc:482] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:34.342985 27453 raft_consensus.cc:3037] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:34.343737 27453 raft_consensus.cc:504] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.343887 27453 leader_election.cc:288] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 election: Requested vote from peers 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:34.344137 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563494cf6c034f3fb35176691e8da9f6" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce"
I20260810 07:24:34.344134 26968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563494cf6c034f3fb35176691e8da9f6" candidate_uuid: "4c6ecafc5f524614aabc750635cc29fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac5d35a259f4961950c3d1b85c35582"
I20260810 07:24:34.344230 27295 raft_consensus.cc:3037] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:34.344247 26968 raft_consensus.cc:3037] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:34.345003 26968 raft_consensus.cc:2454] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 1.
I20260810 07:24:34.345010 27295 raft_consensus.cc:2454] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6ecafc5f524614aabc750635cc29fb in term 1.
I20260810 07:24:34.345250 27053 leader_election.cc:302] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, 8ac5d35a259f4961950c3d1b85c35582; no voters:
I20260810 07:24:34.345371 27453 raft_consensus.cc:2781] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:34.345430 27453 raft_consensus.cc:686] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 1 LEADER]: Becoming Leader. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Running, Role: LEADER
I20260810 07:24:34.345506 27453 consensus_queue.cc:237] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.346203 26776 catalog_manager.cc:5495] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194). New cstate: current_term: 1 leader_uuid: "4c6ecafc5f524614aabc750635cc29fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:34.370486 27485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.367961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.24.60.194:41381)
0810 07:24:34.368183 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.368184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.368194 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:34.368215 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.368216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.368226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.368300 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:34.368440 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.368996 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.369893 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.370102 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.370171 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.370283 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:34.370311 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:34.370350 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.370397 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.370414 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
I20260810 07:24:34.373548 26968 raft_consensus.cc:1261] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Refusing update from remote peer 4c6ecafc5f524614aabc750635cc29fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:34.373539 27295 raft_consensus.cc:1261] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Refusing update from remote peer 4c6ecafc5f524614aabc750635cc29fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:34.373808 27453 consensus_queue.cc:1035] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:34.373953 27383 consensus_queue.cc:1035] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:34.385131 26776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:34.385324 26776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:34.386102 26776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Sending DeleteTablet for 3 replicas of tablet 563494cf6c034f3fb35176691e8da9f6
I20260810 07:24:34.386418 27270 tablet_service.cc:1514] Processing DeleteTablet for tablet 563494cf6c034f3fb35176691e8da9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:34 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:33006
I20260810 07:24:34.386420 27107 tablet_service.cc:1514] Processing DeleteTablet for tablet 563494cf6c034f3fb35176691e8da9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:34 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:55194
I20260810 07:24:34.386463 26944 tablet_service.cc:1514] Processing DeleteTablet for tablet 563494cf6c034f3fb35176691e8da9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:34 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:48032
I20260810 07:24:34.386556 27462 tablet_replica.cc:330] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: stopping tablet replica
I20260810 07:24:34.386595 27464 tablet_replica.cc:330] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: stopping tablet replica
I20260810 07:24:34.386631 27462 raft_consensus.cc:2229] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:34.386642 27464 raft_consensus.cc:2229] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:34.386685 27464 raft_consensus.cc:2258] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:34.386699 27462 raft_consensus.cc:2258] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:34.386798 27463 tablet_replica.cc:330] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: stopping tablet replica
I20260810 07:24:34.386881 27463 raft_consensus.cc:2229] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:34.386940 27463 raft_consensus.cc:2258] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:34.386863 26776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:34.387081 26776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:34.387320 27464 ts_tablet_manager.cc:1905] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:34.387477 27462 ts_tablet_manager.cc:1905] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:34.388993 27463 ts_tablet_manager.cc:1905] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:34.389436 27464 ts_tablet_manager.cc:1918] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:34.389521 27464 log.cc:1196] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/563494cf6c034f3fb35176691e8da9f6
I20260810 07:24:34.389775 27464 ts_tablet_manager.cc:1939] T 563494cf6c034f3fb35176691e8da9f6 P 8ac5d35a259f4961950c3d1b85c35582: Deleting consensus metadata
I20260810 07:24:34.389889 27107 tablet_service.cc:1467] Processing CreateTablet for tablet eedb825e89344870903bd6a3bee1c5ef (DEFAULT_TABLE table=test-table [id=bf092123e42b416aa295299ea3e6b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:34.389889 27270 tablet_service.cc:1467] Processing CreateTablet for tablet eedb825e89344870903bd6a3bee1c5ef (DEFAULT_TABLE table=test-table [id=bf092123e42b416aa295299ea3e6b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:34.390111 27270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedb825e89344870903bd6a3bee1c5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:34.390232 27462 ts_tablet_manager.cc:1918] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:34.390290 27462 log.cc:1196] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/563494cf6c034f3fb35176691e8da9f6
I20260810 07:24:34.390523 27462 ts_tablet_manager.cc:1939] T 563494cf6c034f3fb35176691e8da9f6 P 4c6ecafc5f524614aabc750635cc29fb: Deleting consensus metadata
I20260810 07:24:34.390689 27107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedb825e89344870903bd6a3bee1c5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:34.391098 27463 ts_tablet_manager.cc:1918] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:34.391148 27463 log.cc:1196] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/563494cf6c034f3fb35176691e8da9f6
I20260810 07:24:34.391300 27477 tablet_bootstrap.cc:492] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Bootstrap starting.
I20260810 07:24:34.391360 27463 ts_tablet_manager.cc:1939] T 563494cf6c034f3fb35176691e8da9f6 P d5488ce645ea4a028277f8193b2d05ce: Deleting consensus metadata
I20260810 07:24:34.391671 26765 catalog_manager.cc:4841] TS 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945): tablet 563494cf6c034f3fb35176691e8da9f6 (table test-table [id=1a0aaf3f7063415eb7b78611cf66fe9a]) successfully deleted
I20260810 07:24:34.391693 27477 tablet_bootstrap.cc:654] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:34.391757 26944 tablet_service.cc:1467] Processing CreateTablet for tablet eedb825e89344870903bd6a3bee1c5ef (DEFAULT_TABLE table=test-table [id=bf092123e42b416aa295299ea3e6b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:34.391872 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedb825e89344870903bd6a3bee1c5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:34.392239 27478 tablet_bootstrap.cc:492] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Bootstrap starting.
I20260810 07:24:34.392359 27477 tablet_bootstrap.cc:492] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: No bootstrap required, opened a new log
I20260810 07:24:34.392409 27477 ts_tablet_manager.cc:1397] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:34.392552 27478 tablet_bootstrap.cc:654] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:34.392560 27477 raft_consensus.cc:348] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.392649 27477 raft_consensus.cc:374] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:34.392675 27477 raft_consensus.cc:729] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5488ce645ea4a028277f8193b2d05ce, State: Initialized, Role: FOLLOWER
I20260810 07:24:34.392707 27477 consensus_queue.cc:260] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.392795 27477 ts_tablet_manager.cc:1428] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:34.393036 26766 catalog_manager.cc:4841] TS 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381): tablet 563494cf6c034f3fb35176691e8da9f6 (table test-table [id=1a0aaf3f7063415eb7b78611cf66fe9a]) successfully deleted
I20260810 07:24:34.393257 27476 tablet_bootstrap.cc:492] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582: Bootstrap starting.
I20260810 07:24:34.393494 26765 catalog_manager.cc:4841] TS d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669): tablet 563494cf6c034f3fb35176691e8da9f6 (table test-table [id=1a0aaf3f7063415eb7b78611cf66fe9a]) successfully deleted
I20260810 07:24:34.393714 27476 tablet_bootstrap.cc:654] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:34.393920 27478 tablet_bootstrap.cc:492] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: No bootstrap required, opened a new log
I20260810 07:24:34.393992 27478 ts_tablet_manager.cc:1397] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:34.394114 27478 raft_consensus.cc:348] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.394218 27478 raft_consensus.cc:374] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:34.394246 27478 raft_consensus.cc:729] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6ecafc5f524614aabc750635cc29fb, State: Initialized, Role: FOLLOWER
I20260810 07:24:34.394304 27478 consensus_queue.cc:260] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.394411 27478 ts_tablet_manager.cc:1428] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:34.396322 27476 tablet_bootstrap.cc:492] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582: No bootstrap required, opened a new log
I20260810 07:24:34.396401 27476 ts_tablet_manager.cc:1397] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 07:24:34.396620 27476 raft_consensus.cc:348] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.396739 27476 raft_consensus.cc:374] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:34.396770 27476 raft_consensus.cc:729] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac5d35a259f4961950c3d1b85c35582, State: Initialized, Role: FOLLOWER
I20260810 07:24:34.396827 27476 consensus_queue.cc:260] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.396934 27476 ts_tablet_manager.cc:1428] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:34.420295 27482 raft_consensus.cc:482] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:34.420413 27482 raft_consensus.cc:504] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.420874 27482 leader_election.cc:288] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:34.424765 27494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.421126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:41381 (local address 127.24.60.193:53687)
0810 07:24:34.421401 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:34.421417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:34.421502 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:34.421692 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:34.421693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:34.421696 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:34.421788 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.421790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.422614 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.422615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.423725 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.423728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.424481 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.424482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.424561 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.424567 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:24:34.424624 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 07:24:34.424673 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":26,"thread_start_us":136,"threads_started":1}
I20260810 07:24:34.424818 27485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.421319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53687 (local address 127.24.60.194:41381)
0810 07:24:34.421418 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.421421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.421470 (+ 49us) server_negotiation.cc:407] Connection header received
0810 07:24:34.421588 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.421590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.421603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.421665 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:34.421866 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.422461 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.423830 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.424359 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.424436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.424446 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.424653 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.424702 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 07:24:34.424978 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.421227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:39623 (local address 127.24.60.195:44669)
0810 07:24:34.421345 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.421348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.421495 (+ 147us) server_negotiation.cc:407] Connection header received
0810 07:24:34.421587 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.421588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.421600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.421709 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:34.421943 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.422437 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.424041 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.424532 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.424622 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.424632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.424841 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.424892 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260810 07:24:34.425542 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.421102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:44669 (local address 127.24.60.193:39623)
0810 07:24:34.421384 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:34.421401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:34.421502 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:34.421740 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:34.421741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:34.421744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:34.421848 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.421851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.422543 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.422544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.423946 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.423949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.424658 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.424659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.424746 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.424751 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:34.424827 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 07:24:34.425422 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":45,"thread_start_us":89,"threads_started":1}
I20260810 07:24:34.425668 27133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedb825e89344870903bd6a3bee1c5ef" candidate_uuid: "8ac5d35a259f4961950c3d1b85c35582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6ecafc5f524614aabc750635cc29fb" is_pre_election: true
I20260810 07:24:34.425788 27133 raft_consensus.cc:2454] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac5d35a259f4961950c3d1b85c35582 in term 0.
I20260810 07:24:34.425868 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedb825e89344870903bd6a3bee1c5ef" candidate_uuid: "8ac5d35a259f4961950c3d1b85c35582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce" is_pre_election: true
I20260810 07:24:34.425992 27295 raft_consensus.cc:2454] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac5d35a259f4961950c3d1b85c35582 in term 0.
I20260810 07:24:34.425988 26889 leader_election.cc:302] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, 8ac5d35a259f4961950c3d1b85c35582; no voters:
I20260810 07:24:34.426138 27482 raft_consensus.cc:2781] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:34.426183 27482 raft_consensus.cc:482] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:34.426216 27482 raft_consensus.cc:3037] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:34.427023 27482 raft_consensus.cc:504] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.427205 27482 leader_election.cc:288] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [CANDIDATE]: Term 1 election: Requested vote from peers 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381), d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669)
I20260810 07:24:34.427358 27133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedb825e89344870903bd6a3bee1c5ef" candidate_uuid: "8ac5d35a259f4961950c3d1b85c35582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6ecafc5f524614aabc750635cc29fb"
I20260810 07:24:34.427465 27133 raft_consensus.cc:3037] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:34.427502 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedb825e89344870903bd6a3bee1c5ef" candidate_uuid: "8ac5d35a259f4961950c3d1b85c35582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5488ce645ea4a028277f8193b2d05ce"
I20260810 07:24:34.427639 27295 raft_consensus.cc:3037] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:34.428211 27133 raft_consensus.cc:2454] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac5d35a259f4961950c3d1b85c35582 in term 1.
I20260810 07:24:34.428395 26889 leader_election.cc:302] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6ecafc5f524614aabc750635cc29fb, 8ac5d35a259f4961950c3d1b85c35582; no voters:
I20260810 07:24:34.428412 27295 raft_consensus.cc:2454] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac5d35a259f4961950c3d1b85c35582 in term 1.
I20260810 07:24:34.428544 27482 raft_consensus.cc:2781] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:34.428908 27482 raft_consensus.cc:686] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac5d35a259f4961950c3d1b85c35582, State: Running, Role: LEADER
I20260810 07:24:34.429028 27482 consensus_queue.cc:237] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } }
I20260810 07:24:34.430060 26776 catalog_manager.cc:5495] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "8ac5d35a259f4961950c3d1b85c35582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac5d35a259f4961950c3d1b85c35582" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:34.438424 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.435577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.24.60.193:43945)
0810 07:24:34.435767 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.435768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.435775 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:24:34.435811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.435812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.435832 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.435928 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:34.436052 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.436653 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.437531 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.437773 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.437856 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.438034 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:34.438158 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:34.438247 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.438290 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.438312 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260810 07:24:34.443845 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.441249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:50377 (local address 127.24.60.195:44669)
0810 07:24:34.441314 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.441316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.441444 (+ 128us) server_negotiation.cc:407] Connection header received
0810 07:24:34.441477 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.441478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.441489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.441562 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:34.441737 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.442225 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.443062 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.443266 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.443340 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.443559 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:34.443600 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:34.443665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.443728 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.443756 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 07:24:34.443874 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.441108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:44669 (local address 127.24.60.193:50377)
0810 07:24:34.441343 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:34.441354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:34.441420 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:34.441565 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:34.441566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:34.441568 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:34.441649 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.441651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.442317 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.442318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.442954 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.442956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.443374 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.443375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.443450 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.443469 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:34.443666 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:34.443669 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:34.443723 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 07:24:34.443774 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":132,"threads_started":1}
I20260810 07:24:34.446599 27295 raft_consensus.cc:1261] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Refusing update from remote peer 8ac5d35a259f4961950c3d1b85c35582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:34.446645 27133 raft_consensus.cc:1261] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Refusing update from remote peer 8ac5d35a259f4961950c3d1b85c35582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:34.446884 27482 consensus_queue.cc:1035] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6ecafc5f524614aabc750635cc29fb" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:34.447260 27496 consensus_queue.cc:1035] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5488ce645ea4a028277f8193b2d05ce" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:34.455272 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.452507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43945 (local address 127.24.60.195:55891)
0810 07:24:34.452673 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:34.452686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:34.452739 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:34.452859 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:34.452860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:34.452863 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:34.452930 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.452932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.453750 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.453751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.454363 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:34.454366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.454838 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.454840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:34.454904 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.454910 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:34.455105 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:34.455107 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 07:24:34.455155 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:24:34.455209 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260810 07:24:34.455272 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:34.452696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:55891 (local address 127.24.60.193:43945)
0810 07:24:34.452739 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:34.452741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:34.452750 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:34.452782 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:34.452783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:34.452792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:34.452855 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:34.452988 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.453660 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.454460 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:34.454741 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:34.454834 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:34.454990 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:34.455027 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:34.455078 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:34.455164 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 07:24:34.455193 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 07:24:34.455657 27442 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 07:24:35.976322 26773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:35.976460 26773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:35.977491 26773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cd49c1e28dd4b5085aca4ff4d8df207: Sending DeleteTablet for 3 replicas of tablet eedb825e89344870903bd6a3bee1c5ef
I20260810 07:24:35.977790 26944 tablet_service.cc:1514] Processing DeleteTablet for tablet eedb825e89344870903bd6a3bee1c5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:35 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:48032
I20260810 07:24:35.977873 27270 tablet_service.cc:1514] Processing DeleteTablet for tablet eedb825e89344870903bd6a3bee1c5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:35 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:33006
I20260810 07:24:35.978031 27538 tablet_replica.cc:330] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582: stopping tablet replica
I20260810 07:24:35.978025 27107 tablet_service.cc:1514] Processing DeleteTablet for tablet eedb825e89344870903bd6a3bee1c5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:35 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:55194
I20260810 07:24:35.978101 27538 raft_consensus.cc:2229] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:35.978225 27538 raft_consensus.cc:2258] T eedb825e89344870903bd6a3bee1c5ef P 8ac5d35a259f4961950c3d1b85c35582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:35.978256 27541 tablet_replica.cc:330] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: stopping tablet replica
I20260810 07:24:35.978320 27541 raft_consensus.cc:2229] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:35.978348 27540 tablet_replica.cc:330] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: stopping tablet replica
I20260810 07:24:35.978469 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 26869
I20260810 07:24:35.978394 27541 raft_consensus.cc:2258] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:35.991704 27541 ts_tablet_manager.cc:1905] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:35.994132 27541 ts_tablet_manager.cc:1918] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:24:35.994196 27541 log.cc:1196] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/eedb825e89344870903bd6a3bee1c5ef
I20260810 07:24:35.994451 27541 ts_tablet_manager.cc:1939] T eedb825e89344870903bd6a3bee1c5ef P 4c6ecafc5f524614aabc750635cc29fb: Deleting consensus metadata
I20260810 07:24:35.994899 26766 catalog_manager.cc:4841] TS 4c6ecafc5f524614aabc750635cc29fb (127.24.60.194:41381): tablet eedb825e89344870903bd6a3bee1c5ef (table test-table [id=bf092123e42b416aa295299ea3e6b022]) successfully deleted
I20260810 07:24:35.995483 27540 raft_consensus.cc:2229] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:35.995576 27540 raft_consensus.cc:2258] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:35.995896 27540 ts_tablet_manager.cc:1905] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:35.997588 27540 ts_tablet_manager.cc:1918] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:24:35.997643 27540 log.cc:1196] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/eedb825e89344870903bd6a3bee1c5ef
I20260810 07:24:35.997882 27540 ts_tablet_manager.cc:1939] T eedb825e89344870903bd6a3bee1c5ef P d5488ce645ea4a028277f8193b2d05ce: Deleting consensus metadata
I20260810 07:24:35.998461 26765 catalog_manager.cc:4841] TS d5488ce645ea4a028277f8193b2d05ce (127.24.60.195:44669): tablet eedb825e89344870903bd6a3bee1c5ef (table test-table [id=bf092123e42b416aa295299ea3e6b022]) successfully deleted
W20260810 07:24:35.999675 26765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:43945 (error 108)
I20260810 07:24:36.000401 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 27033
I20260810 07:24:36.000447 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:36.000172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43945 (local address 127.0.0.1:55252)
0810 07:24:36.000333 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:36.000395 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
W20260810 07:24:36.003403 26765 catalog_manager.cc:4584] TS 8ac5d35a259f4961950c3d1b85c35582 (127.24.60.193:43945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eedb825e89344870903bd6a3bee1c5ef: Network error: Client connection negotiation failed: client connection to 127.24.60.193:43945: connect: Connection refused (error 111)
I20260810 07:24:36.015502 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 27191
I20260810 07:24:36.025112 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 26749
2026-08-10T07:24:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6088 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:36 dist-test-slave-vxj2 krb5kdc[27548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:36 dist-test-slave-vxj2 krb5kdc[27548](info): set up 2 sockets
Aug 10 07:24:36 dist-test-slave-vxj2 krb5kdc[27548](info): commencing operation
krb5kdc: starting...
W20260810 07:24:37.161476 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:37.296581 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33501
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:37.387069 27563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:37.387408 27563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:37.387465 27563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:37.389562 27563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:37.389645 27563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:37.389673 27563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:37.389699 27563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:37.389732 27563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:37.392439 27563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:37.392812 27563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:37.393095 27563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:37.399173 27571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:37.399183 27573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.399170 27563 server_base.cc:1034] running on GCE node
W20260810 07:24:37.399442 27569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.399792 27563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:37.400153 27563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:37.401324 27563 hybrid_clock.cc:648] HybridClock initialized: now 1786346677401314 us; error 38 us; skew 500 ppm
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:37.408159 27563 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:37.408829 27563 webserver.cc:460] Webserver started at http://127.24.60.254:41337/ using document root <none> and password file <none>
I20260810 07:24:37.409093 27563 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:37.409194 27563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:37.409374 27563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:37.410573 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "10f629348d3540738ad6806a1f2f3a3a"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "33981b54d71f435b71ffb27ebd44ccdc"
server_key_iv: "b09697a4e5c5f0b6d221bb31176b3752"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.410984 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "10f629348d3540738ad6806a1f2f3a3a"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "33981b54d71f435b71ffb27ebd44ccdc"
server_key_iv: "b09697a4e5c5f0b6d221bb31176b3752"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.412606 27563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:37.413564 27582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.413769 27563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:37.413884 27563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "10f629348d3540738ad6806a1f2f3a3a"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "33981b54d71f435b71ffb27ebd44ccdc"
server_key_iv: "b09697a4e5c5f0b6d221bb31176b3752"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.413967 27563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:37.431339 27563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:37.433604 27563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:37.433775 27563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:37.438361 27563 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:33501
I20260810 07:24:37.438402 27640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:33501 every 8 connection(s)
I20260810 07:24:37.438829 27563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:37.439859 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:37.442838 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 27563
I20260810 07:24:37.442920 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:37.442862 27641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a: Bootstrap starting.
I20260810 07:24:37.443106 24819 external_mini_cluster.cc:1428] Setting key 19b2317efd3569715bd59854976ee6f6
I20260810 07:24:37.443832 27641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:37.444221 27641 log.cc:824] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:37.445115 27641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a: No bootstrap required, opened a new log
I20260810 07:24:37.447311 27641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f629348d3540738ad6806a1f2f3a3a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33501 } }
I20260810 07:24:37.447516 27641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:37.447567 27641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f629348d3540738ad6806a1f2f3a3a, State: Initialized, Role: FOLLOWER
I20260810 07:24:37.447738 27641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f629348d3540738ad6806a1f2f3a3a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33501 } }
I20260810 07:24:37.447867 27641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:37.447925 27641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:37.447986 27641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:37.448832 27641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f629348d3540738ad6806a1f2f3a3a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33501 } }
I20260810 07:24:37.449013 27641 leader_election.cc:302] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10f629348d3540738ad6806a1f2f3a3a; no voters:
I20260810 07:24:37.449322 27641 leader_election.cc:288] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:37.449366 27647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:37.449537 27647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [term 1 LEADER]: Becoming Leader. State: Replica: 10f629348d3540738ad6806a1f2f3a3a, State: Running, Role: LEADER
I20260810 07:24:37.449654 27641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:37.449651 27647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f629348d3540738ad6806a1f2f3a3a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33501 } }
I20260810 07:24:37.450148 27649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f629348d3540738ad6806a1f2f3a3a. Latest consensus state: current_term: 1 leader_uuid: "10f629348d3540738ad6806a1f2f3a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f629348d3540738ad6806a1f2f3a3a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33501 } } }
I20260810 07:24:37.450242 27649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [sys.catalog]: This master's current role is: LEADER
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:37.450991 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10f629348d3540738ad6806a1f2f3a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f629348d3540738ad6806a1f2f3a3a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33501 } } }
I20260810 07:24:37.451068 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:37.451207 27652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:37.451858 27652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:37.453714 27652 catalog_manager.cc:1347] Generated new cluster ID: 87891e6ef0544f9ea0b933b3fb6ad730
I20260810 07:24:37.453796 27652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:37.455583 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:37.443799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.24.60.254:33501)
0810 07:24:37.444113 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:37.444121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:37.444148 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:24:37.444684 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:37.444691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:37.444755 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:37.444915 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:37.445687 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.446607 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.447403 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.448166 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.448291 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:37.451535 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:37.451541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:37.451542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:37.451556 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:37.453975 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:37.454360 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:37.454363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:37.454364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:37.454459 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:37.454685 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:37.454686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:37.454686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:37.454940 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:37.455123 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:37.455247 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 07:24:37.455401 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260810 07:24:37.461838 27652 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:37.462633 27652 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:37.467537 27652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10f629348d3540738ad6806a1f2f3a3a: Generated new TSK 0
I20260810 07:24:37.467756 27652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:37.504994 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33501
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:37.623067 27670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:37.623286 27670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:37.623314 27670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:37.624534 27670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:37.624577 27670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:37.624605 27670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:37.626163 27670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:37.626435 27670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:37.626771 27670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:37.632288 27678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:37.632273 27677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:37.632513 27681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.632843 27670 server_base.cc:1034] running on GCE node
I20260810 07:24:37.633111 27670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:37.633494 27670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:37.634704 27670 hybrid_clock.cc:648] HybridClock initialized: now 1786346677634687 us; error 27 us; skew 500 ppm
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:37.638199 27670 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:37.638796 27670 webserver.cc:460] Webserver started at http://127.24.60.193:46607/ using document root <none> and password file <none>
I20260810 07:24:37.639073 27670 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:37.639147 27670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:37.639308 27670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:37.640391 27670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "bfcce5aed1ab404c922239eb29996976"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "cb7ff1579249bd1816178aa72d739cc4"
server_key_iv: "96e1b516fc8f4b5a5549cc0afe851ac8"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.640771 27670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "bfcce5aed1ab404c922239eb29996976"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "cb7ff1579249bd1816178aa72d739cc4"
server_key_iv: "96e1b516fc8f4b5a5549cc0afe851ac8"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.642431 27670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:24:37.643342 27688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.643581 27670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:37.643702 27670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "bfcce5aed1ab404c922239eb29996976"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "cb7ff1579249bd1816178aa72d739cc4"
server_key_iv: "96e1b516fc8f4b5a5549cc0afe851ac8"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.643800 27670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:37.665597 27670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:37.668089 27670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:37.668223 27670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:37.668447 27670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:37.668882 27670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:37.668927 27670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.668951 27670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:37.668964 27670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.675136 27670 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:34769
I20260810 07:24:37.675226 27820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:34769 every 8 connection(s)
I20260810 07:24:37.675575 27670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:37.682384 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 27670
I20260810 07:24:37.682524 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:37.682754 24819 external_mini_cluster.cc:1428] Setting key e155db7db86397323c3da08d0759b6ee
I20260810 07:24:37.686061 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:37.676412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:54235 (local address 127.24.60.254:33501)
0810 07:24:37.676512 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:37.676514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:37.676714 (+ 200us) server_negotiation.cc:407] Connection header received
0810 07:24:37.677276 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:37.677277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:37.677287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:37.677341 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:37.678289 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.678796 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.679838 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.680053 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.680112 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:37.682494 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:37.682497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:37.682498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:37.682509 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:37.684405 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:37.684890 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:37.684892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:37.684892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:37.684963 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:37.685323 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:37.685324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:37.685324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:37.685456 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:37.685538 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:37.685853 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 07:24:37.685971 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:24:37.686265 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:37.676387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33501 (local address 127.24.60.193:54235)
0810 07:24:37.676585 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:37.676624 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:37.677182 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:37.677396 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:37.677399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:37.677989 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:37.678204 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:37.678209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.678888 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.678888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:37.679746 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:37.679749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.680139 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.680140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:37.680247 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:37.680746 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:37.680755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:37.682365 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:37.684524 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:37.684534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:37.684537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:37.684803 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:37.685044 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:37.685045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:37.685046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:37.685245 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:37.685529 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:37.685537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:37.685692 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 07:24:37.685816 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 07:24:37.686001 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260810 07:24:37.686911 27821 heartbeater.cc:347] Connected to a master server at 127.24.60.254:33501
I20260810 07:24:37.687074 27821 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:37.687299 27821 heartbeater.cc:511] Master 127.24.60.254:33501 requested a full tablet report, sending...
I20260810 07:24:37.687917 27602 ts_manager.cc:194] Registered new tserver with Master: bfcce5aed1ab404c922239eb29996976 (127.24.60.193:34769)
I20260810 07:24:37.688927 27602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:54235
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:37.712769 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33501
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:37.799161 27827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:37.799396 27827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:37.799425 27827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:37.800778 27827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:37.800833 27827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:37.800899 27827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:37.802603 27827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:37.802911 27827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:37.803175 27827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:37.808736 27834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:37.808795 27833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.808799 27827 server_base.cc:1034] running on GCE node
W20260810 07:24:37.808763 27837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.809327 27827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:37.809602 27827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:37.810791 27827 hybrid_clock.cc:648] HybridClock initialized: now 1786346677810749 us; error 55 us; skew 500 ppm
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:37.813655 27827 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:37.814199 27827 webserver.cc:460] Webserver started at http://127.24.60.194:46015/ using document root <none> and password file <none>
I20260810 07:24:37.814473 27827 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:37.814580 27827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:37.814754 27827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:37.815918 27827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "ac6d380e160847e6a342350b0b176fd5"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "ccae105955fba83d584696e1c6ecb666"
server_key_iv: "82dc35e0ae69cad95a49e9ea45f0550f"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.816326 27827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "ac6d380e160847e6a342350b0b176fd5"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "ccae105955fba83d584696e1c6ecb666"
server_key_iv: "82dc35e0ae69cad95a49e9ea45f0550f"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.817889 27827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:37.818755 27844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.818948 27827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:37.819099 27827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "ac6d380e160847e6a342350b0b176fd5"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "ccae105955fba83d584696e1c6ecb666"
server_key_iv: "82dc35e0ae69cad95a49e9ea45f0550f"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.819178 27827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:37.826007 27827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:37.828402 27827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:37.828575 27827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:37.828825 27827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:37.829222 27827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:37.829262 27827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.829286 27827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:37.829330 27827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.835741 27827 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:36203
I20260810 07:24:37.835788 27974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:36203 every 8 connection(s)
I20260810 07:24:37.836153 27827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:37.838704 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 27827
I20260810 07:24:37.838822 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:37.839030 24819 external_mini_cluster.cc:1428] Setting key e6843a737fd18217726cbccbecc69c4c
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:37.848708 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:37.837113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:60523 (local address 127.24.60.254:33501)
0810 07:24:37.837224 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:37.837227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:37.837392 (+ 165us) server_negotiation.cc:407] Connection header received
0810 07:24:37.837908 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:37.837909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:37.837928 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:37.838015 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:37.839438 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.840136 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.841854 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.842184 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.842269 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:37.845131 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:37.845133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:37.845133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:37.845141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:37.846758 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:37.847394 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:37.847395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:37.847396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:37.847479 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:37.847833 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:37.847834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:37.847834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:37.847953 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:37.848017 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:37.848490 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 07:24:37.848602 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
I20260810 07:24:37.849422 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:37.837067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33501 (local address 127.24.60.194:60523)
0810 07:24:37.837259 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:37.837303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:37.837815 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:37.838096 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:37.838100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:37.838923 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:37.839282 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:37.839310 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.840253 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.840254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:37.841751 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:37.841754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:37.842282 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:37.842283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:37.842400 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:37.843034 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:37.843043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:37.844944 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:37.846868 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:37.846880 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:37.846882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:37.847286 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:37.847569 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:37.847570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:37.847570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:37.847751 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:37.848034 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:37.848043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:37.848256 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 07:24:37.848836 (+ 580us) client_negotiation.cc:240] Negotiation successful
0810 07:24:37.849087 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260810 07:24:37.850019 27975 heartbeater.cc:347] Connected to a master server at 127.24.60.254:33501
I20260810 07:24:37.850145 27975 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:37.850329 27975 heartbeater.cc:511] Master 127.24.60.254:33501 requested a full tablet report, sending...
I20260810 07:24:37.850907 27602 ts_manager.cc:194] Registered new tserver with Master: ac6d380e160847e6a342350b0b176fd5 (127.24.60.194:36203)
I20260810 07:24:37.851382 27602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:60523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:37.869407 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33501
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:37.957144 27981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:37.957466 27981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:37.957494 27981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:37.958784 27981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:37.958837 27981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:37.958865 27981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:37.960403 27981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:37.960690 27981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:37.960954 27981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:37.966830 27988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.966912 27981 server_base.cc:1034] running on GCE node
W20260810 07:24:37.966795 27990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:37.966856 27987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:37.967342 27981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:37.967705 27981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:37.969051 27981 hybrid_clock.cc:648] HybridClock initialized: now 1786346677969023 us; error 44 us; skew 500 ppm
Aug 10 07:24:37 dist-test-slave-vxj2 krb5kdc[27548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:37.971998 27981 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:37.972510 27981 webserver.cc:460] Webserver started at http://127.24.60.195:40809/ using document root <none> and password file <none>
I20260810 07:24:37.972772 27981 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:37.972855 27981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:37.973019 27981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:37.974279 27981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "56047eedacdc40c4899446230fc16420"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "5207d3215f0df9a7c0497cd8f9a612c2"
server_key_iv: "afb1a49e8c0043120dde389e8dd8dbeb"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.974684 27981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "56047eedacdc40c4899446230fc16420"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "5207d3215f0df9a7c0497cd8f9a612c2"
server_key_iv: "afb1a49e8c0043120dde389e8dd8dbeb"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.976245 27981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:37.977265 27998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:37.977468 27981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:37.977588 27981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "56047eedacdc40c4899446230fc16420"
format_stamp: "Formatted at 2026-08-10 07:24:37 on dist-test-slave-vxj2"
server_key: "5207d3215f0df9a7c0497cd8f9a612c2"
server_key_iv: "afb1a49e8c0043120dde389e8dd8dbeb"
server_key_version: "encryptionkey@0"
I20260810 07:24:37.977675 27981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:37.999619 27981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:38.002388 27981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:38.002594 27981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:38.002884 27981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:38.003247 27981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:38.003304 27981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:38.003343 27981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:38.003392 27981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:38.009981 27981 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:34277
I20260810 07:24:38.010130 28128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:34277 every 8 connection(s)
I20260810 07:24:38.010460 27981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:38.015625 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 27981
I20260810 07:24:38.015764 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:38.015975 24819 external_mini_cluster.cc:1428] Setting key 782df90b7527d38dea6356f2d38c38e8
Aug 10 07:24:38 dist-test-slave-vxj2 krb5kdc[27548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:38.021313 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:38.011400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:59227 (local address 127.24.60.254:33501)
0810 07:24:38.011471 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:38.011473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:38.011680 (+ 207us) server_negotiation.cc:407] Connection header received
0810 07:24:38.012176 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:38.012178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:38.012188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:38.012252 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:38.013259 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:38.013798 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:38.014860 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:38.015055 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:38.015119 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:38.018117 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:38.018118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:38.018119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:38.018127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:38.019537 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:38.020174 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:38.020176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:38.020176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:38.020242 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:38.020540 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:38.020541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:38.020541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:38.020676 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:38.020755 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:38.021071 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 07:24:38.021206 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:24:38.021514 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:38.011372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33501 (local address 127.24.60.195:59227)
0810 07:24:38.011546 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:38.011589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:38.012088 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:38.012306 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:38.012310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:38.012905 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:38.013117 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:38.013122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:38.013893 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:38.013894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:38.014769 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:38.014772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:38.015146 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:38.015147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:38.015253 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:38.015933 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:38.015942 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:38.017998 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:38.019755 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:38.019769 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:38.019772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:38.020086 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:38.020339 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:38.020340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:38.020341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:38.020464 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:38.020757 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:38.020766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:38.020924 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 07:24:38.021036 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 07:24:38.021248 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260810 07:24:38.022064 28129 heartbeater.cc:347] Connected to a master server at 127.24.60.254:33501
I20260810 07:24:38.022181 28129 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:38.022414 28129 heartbeater.cc:511] Master 127.24.60.254:33501 requested a full tablet report, sending...
I20260810 07:24:38.022929 27602 ts_manager.cc:194] Registered new tserver with Master: 56047eedacdc40c4899446230fc16420 (127.24.60.195:34277)
I20260810 07:24:38.023376 27602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:59227
I20260810 07:24:38.028434 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 07:24:38.041380 27644 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:38.040278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.24.60.254:33501)
0810 07:24:38.040373 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:38.040375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:38.040470 (+ 95us) server_negotiation.cc:407] Connection header received
0810 07:24:38.040535 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:38.040536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:38.040547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:38.040601 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:38.041277 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49960: BlockingRecv error: recv got EOF from 127.0.0.1:49960 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:24:38.042174 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 27670
I20260810 07:24:38.049818 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 27827
I20260810 07:24:38.058156 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 27981
I20260810 07:24:38.065034 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 27563
2026-08-10T07:24:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2036 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:38 dist-test-slave-vxj2 krb5kdc[28143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:38 dist-test-slave-vxj2 krb5kdc[28143](info): set up 2 sockets
Aug 10 07:24:38 dist-test-slave-vxj2 krb5kdc[28143](info): commencing operation
krb5kdc: starting...
W20260810 07:24:39.219945 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:39 dist-test-slave-vxj2 krb5kdc[28143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:39.356549 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36823
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:36823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:39.448115 28157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:39.448434 28157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:39.448482 28157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:39.449808 28157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:39.449898 28157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:39.449923 28157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:39.449946 28157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:39.449973 28157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:39.451753 28157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:36823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:39.452077 28157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:39.452342 28157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:39.457937 28162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:39.457928 28163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:39.458166 28157 server_base.cc:1034] running on GCE node
W20260810 07:24:39.457927 28165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:39.458621 28157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:39.459021 28157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:39.460222 28157 hybrid_clock.cc:648] HybridClock initialized: now 1786346679460205 us; error 66 us; skew 500 ppm
Aug 10 07:24:39 dist-test-slave-vxj2 krb5kdc[28143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346679, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:39.463230 28157 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:39.463764 28157 webserver.cc:460] Webserver started at http://127.24.60.254:42161/ using document root <none> and password file <none>
I20260810 07:24:39.464025 28157 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:39.464073 28157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:39.464170 28157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:39.465361 28157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "f03139b36d1844e1909bfbcd5ba6a754"
format_stamp: "Formatted at 2026-08-10 07:24:39 on dist-test-slave-vxj2"
server_key: "f29ad79b10cbb9cdfe0403d444e57342"
server_key_iv: "e7a45419698589faf4104094db9a9a42"
server_key_version: "encryptionkey@0"
I20260810 07:24:39.465704 28157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "f03139b36d1844e1909bfbcd5ba6a754"
format_stamp: "Formatted at 2026-08-10 07:24:39 on dist-test-slave-vxj2"
server_key: "f29ad79b10cbb9cdfe0403d444e57342"
server_key_iv: "e7a45419698589faf4104094db9a9a42"
server_key_version: "encryptionkey@0"
I20260810 07:24:39.467252 28157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:39.468120 28172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:39.468420 28157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:39.468549 28157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "f03139b36d1844e1909bfbcd5ba6a754"
format_stamp: "Formatted at 2026-08-10 07:24:39 on dist-test-slave-vxj2"
server_key: "f29ad79b10cbb9cdfe0403d444e57342"
server_key_iv: "e7a45419698589faf4104094db9a9a42"
server_key_version: "encryptionkey@0"
I20260810 07:24:39.468636 28157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:39.482272 28157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:39.485359 28157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:39.485548 28157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:39.490751 28157 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:36823
I20260810 07:24:39.490795 28234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:36823 every 8 connection(s)
I20260810 07:24:39.491354 28157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:39.492199 28235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:39.492957 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28157
I20260810 07:24:39.493072 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:39.493309 24819 external_mini_cluster.cc:1428] Setting key d8b0fdb13ae193e7d42e29fe6ecf5968
I20260810 07:24:39.495937 28235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754: Bootstrap starting.
I20260810 07:24:39.496876 28235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:39.497279 28235 log.cc:824] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:39.498214 28235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754: No bootstrap required, opened a new log
I20260810 07:24:39.500816 28235 raft_consensus.cc:348] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03139b36d1844e1909bfbcd5ba6a754" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36823 } }
I20260810 07:24:39.500986 28235 raft_consensus.cc:374] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:39.501012 28235 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03139b36d1844e1909bfbcd5ba6a754, State: Initialized, Role: FOLLOWER
I20260810 07:24:39.501417 28235 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03139b36d1844e1909bfbcd5ba6a754" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36823 } }
I20260810 07:24:39.501533 28235 raft_consensus.cc:388] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:39.501588 28235 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:39.501649 28235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 07:24:39 dist-test-slave-vxj2 krb5kdc[28143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:39.502419 28235 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03139b36d1844e1909bfbcd5ba6a754" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36823 } }
I20260810 07:24:39.502559 28235 leader_election.cc:302] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f03139b36d1844e1909bfbcd5ba6a754; no voters:
I20260810 07:24:39.502820 28235 leader_election.cc:288] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:39.502863 28242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:39.502991 28242 raft_consensus.cc:686] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [term 1 LEADER]: Becoming Leader. State: Replica: f03139b36d1844e1909bfbcd5ba6a754, State: Running, Role: LEADER
I20260810 07:24:39.503099 28242 consensus_queue.cc:237] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03139b36d1844e1909bfbcd5ba6a754" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36823 } }
I20260810 07:24:39.503153 28235 sys_catalog.cc:564] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:39.503477 28242 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f03139b36d1844e1909bfbcd5ba6a754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03139b36d1844e1909bfbcd5ba6a754" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36823 } } }
I20260810 07:24:39.503561 28242 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:39.503700 28242 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03139b36d1844e1909bfbcd5ba6a754. Latest consensus state: current_term: 1 leader_uuid: "f03139b36d1844e1909bfbcd5ba6a754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03139b36d1844e1909bfbcd5ba6a754" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36823 } } }
I20260810 07:24:39.503793 28242 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:39.504681 28249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:39.505370 28249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:39.507280 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:39.494008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.24.60.254:36823)
0810 07:24:39.494244 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:39.494255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:39.494342 (+ 87us) server_negotiation.cc:407] Connection header received
0810 07:24:39.495026 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:39.495038 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:39.495121 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:39.495311 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:39.496060 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:39.497653 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:39.498801 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:39.499486 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:39.499617 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:39.502778 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:39.502782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:39.502783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:39.502796 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:39.505330 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:39.505792 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:39.505795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:39.505796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:39.506009 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:39.506245 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:39.506246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:39.506246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:39.506589 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:39.506775 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:39.506921 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 07:24:39.507078 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260810 07:24:39.507851 28249 catalog_manager.cc:1347] Generated new cluster ID: ebce6e1068b2417facf1c947662bd09f
I20260810 07:24:39.507906 28249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:39.508330 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:24:39.526954 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:39.522098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.24.60.254:36823)
0810 07:24:39.522150 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:39.522153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:39.522240 (+ 87us) server_negotiation.cc:407] Connection header received
0810 07:24:39.522301 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:39.522302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:39.522315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:39.522412 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:39.522866 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:39.523807 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:39.524905 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:39.525337 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:39.525435 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:39.526297 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:39.526299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:39.526299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:39.526607 (+ 308us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:39.526633 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:39.526642 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:39.526715 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:39.526809 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 07:24:39.526850 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:39.528335 28249 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:39.528919 28249 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:39.533195 28249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f03139b36d1844e1909bfbcd5ba6a754: Generated new TSK 0
I20260810 07:24:39.533484 28249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:24:39.540004 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28157
2026-08-10T07:24:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1475 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:39 dist-test-slave-vxj2 krb5kdc[28278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:39 dist-test-slave-vxj2 krb5kdc[28278](info): set up 2 sockets
Aug 10 07:24:39 dist-test-slave-vxj2 krb5kdc[28278](info): commencing operation
krb5kdc: starting...
W20260810 07:24:40.723145 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:40 dist-test-slave-vxj2 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:40.857587 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:41979
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:40.943148 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:40.943375 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:40.943400 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:40.944541 28292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:40.944587 28292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:40.944602 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:40.944615 28292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:40.944628 28292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:40.946211 28292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:41979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:40.946432 28292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:40.946641 28292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:40.951946 28298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:40.951887 28301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:40.951977 28292 server_base.cc:1034] running on GCE node
W20260810 07:24:40.951911 28299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:40.952508 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:40.952870 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:40.954053 28292 hybrid_clock.cc:648] HybridClock initialized: now 1786346680954000 us; error 70 us; skew 500 ppm
Aug 10 07:24:40 dist-test-slave-vxj2 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346680, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:40.956779 28292 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:40.957338 28292 webserver.cc:460] Webserver started at http://127.24.60.254:44773/ using document root <none> and password file <none>
I20260810 07:24:40.957615 28292 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:40.957690 28292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:40.957861 28292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:40.959025 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "335e4ff2a1164a4085bfd9f34092a9e7"
format_stamp: "Formatted at 2026-08-10 07:24:40 on dist-test-slave-vxj2"
server_key: "b70bea18f5d5240fb622613dabbbfd46"
server_key_iv: "6597e2d8ca5d0058a59609869d284901"
server_key_version: "encryptionkey@0"
I20260810 07:24:40.959425 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "335e4ff2a1164a4085bfd9f34092a9e7"
format_stamp: "Formatted at 2026-08-10 07:24:40 on dist-test-slave-vxj2"
server_key: "b70bea18f5d5240fb622613dabbbfd46"
server_key_iv: "6597e2d8ca5d0058a59609869d284901"
server_key_version: "encryptionkey@0"
I20260810 07:24:40.960803 28292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:40.961735 28308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:40.961964 28292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:24:40.962076 28292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "335e4ff2a1164a4085bfd9f34092a9e7"
format_stamp: "Formatted at 2026-08-10 07:24:40 on dist-test-slave-vxj2"
server_key: "b70bea18f5d5240fb622613dabbbfd46"
server_key_iv: "6597e2d8ca5d0058a59609869d284901"
server_key_version: "encryptionkey@0"
I20260810 07:24:40.962165 28292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:40.975340 28292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:40.978084 28292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:40.978271 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:40.982759 28292 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:41979
I20260810 07:24:40.982976 28371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:41979 every 8 connection(s)
I20260810 07:24:40.983245 28292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:40.983565 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28292
I20260810 07:24:40.983682 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:40.983879 24819 external_mini_cluster.cc:1428] Setting key 9d21c032dfff0e259c084b178191d76c
I20260810 07:24:40.984459 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:40.988090 28372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7: Bootstrap starting.
I20260810 07:24:40.988793 28372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:40.989123 28372 log.cc:824] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:40.990032 28372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7: No bootstrap required, opened a new log
Aug 10 07:24:40 dist-test-slave-vxj2 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:40.992089 28372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41979 } }
I20260810 07:24:40.992264 28372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:40.992313 28372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335e4ff2a1164a4085bfd9f34092a9e7, State: Initialized, Role: FOLLOWER
I20260810 07:24:40.992515 28372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41979 } }
I20260810 07:24:40.992637 28372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:40.992681 28372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:40.992746 28372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:40.993564 28372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41979 } }
I20260810 07:24:40.993661 28372 leader_election.cc:302] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 335e4ff2a1164a4085bfd9f34092a9e7; no voters:
I20260810 07:24:40.993898 28372 leader_election.cc:288] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:40.993961 28377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:40.994143 28377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [term 1 LEADER]: Becoming Leader. State: Replica: 335e4ff2a1164a4085bfd9f34092a9e7, State: Running, Role: LEADER
I20260810 07:24:40.994202 28372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:40.994351 28377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41979 } }
I20260810 07:24:40.994804 28377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335e4ff2a1164a4085bfd9f34092a9e7. Latest consensus state: current_term: 1 leader_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41979 } } }
I20260810 07:24:40.994882 28377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:40.995087 28378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e4ff2a1164a4085bfd9f34092a9e7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41979 } } }
I20260810 07:24:40.995183 28378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:40.995402 28383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:40.996280 28383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:40.996618 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:40.984496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.24.60.254:41979)
0810 07:24:40.984703 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:40.984713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:40.984748 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:24:40.985426 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:40.985437 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:40.985514 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:40.985688 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:40.986377 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:40.987825 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:40.988777 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:40.989464 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:40.989608 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:40.992080 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:40.992085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:40.992085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:40.992099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:40.994145 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:40.994654 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:40.994657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:40.994658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:40.994772 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:40.995022 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:40.995024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:40.995024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:40.995449 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:40.996123 (+ 674us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:40.996199 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 07:24:40.996363 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260810 07:24:40.998003 28383 catalog_manager.cc:1347] Generated new cluster ID: d101b2f0580f4f1ead5ebeccc4ba616c
I20260810 07:24:40.998064 28383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:41.012986 28383 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:41.013819 28383 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:41.016515 28383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 335e4ff2a1164a4085bfd9f34092a9e7: Generated new TSK 0
I20260810 07:24:41.016748 28383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:41.048539 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41979
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:41.135964 28401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:41.136234 28401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:41.136282 28401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:41.137596 28401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:41.137671 28401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:41.137719 28401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:41.139343 28401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:41.139657 28401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:41.139915 28401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:41.145342 28407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:41.145337 28409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:41.145341 28412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:41.145437 28401 server_base.cc:1034] running on GCE node
I20260810 07:24:41.145861 28401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:41.146158 28401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:41.147344 28401 hybrid_clock.cc:648] HybridClock initialized: now 1786346681147306 us; error 54 us; skew 500 ppm
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:41.150954 28401 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:41.151590 28401 webserver.cc:460] Webserver started at http://127.24.60.193:35185/ using document root <none> and password file <none>
I20260810 07:24:41.151892 28401 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:41.151975 28401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:41.152156 28401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:41.153524 28401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "e7a25bb9568f4fb3878fd6ea89593a1e"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "d035aaf52eb1eb8551a916eb7d94cf7f"
server_key_iv: "be72267b545df70f779f1fb4236caeb7"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.153980 28401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a25bb9568f4fb3878fd6ea89593a1e"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "d035aaf52eb1eb8551a916eb7d94cf7f"
server_key_iv: "be72267b545df70f779f1fb4236caeb7"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.155820 28401 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:24:41.156849 28419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.157101 28401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:41.157243 28401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "e7a25bb9568f4fb3878fd6ea89593a1e"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "d035aaf52eb1eb8551a916eb7d94cf7f"
server_key_iv: "be72267b545df70f779f1fb4236caeb7"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.157337 28401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:41.177420 28401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:41.180115 28401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:41.180348 28401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:41.180646 28401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:41.181048 28401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:41.181114 28401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.181154 28401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:41.181236 28401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.188127 28401 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:39921
I20260810 07:24:41.188181 28551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:39921 every 8 connection(s)
I20260810 07:24:41.188568 28401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:41.195303 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28401
I20260810 07:24:41.195431 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:41.195716 24819 external_mini_cluster.cc:1428] Setting key fa1f80df049bc1af7b833cc157bee555
I20260810 07:24:41.200139 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.189441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:40003 (local address 127.24.60.254:41979)
0810 07:24:41.189520 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:41.189522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:41.189774 (+ 252us) server_negotiation.cc:407] Connection header received
0810 07:24:41.190365 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:41.190366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:41.190376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:41.190436 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:41.191510 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.192051 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.193198 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.193400 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.193455 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.196206 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:41.196207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:41.196208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:41.196215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:41.198105 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.198804 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.198806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.198806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.198871 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.199210 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.199211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.199211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.199327 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:41.199382 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:41.199820 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 07:24:41.199946 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:24:41.200171 28556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.189423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41979 (local address 127.24.60.193:40003)
0810 07:24:41.189622 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:41.189672 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:41.190267 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:41.190511 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:41.190515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:41.191178 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:41.191406 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:41.191411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.192153 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.192154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:41.193082 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:41.193084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.193488 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.193489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:41.193626 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.194192 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:41.194201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:41.196111 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:41.198212 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:41.198227 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:41.198230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:41.198700 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:41.198949 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:41.198950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:41.198950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:41.199132 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:41.199403 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:41.199412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:41.199643 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 07:24:41.199797 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 07:24:41.199974 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
I20260810 07:24:41.200800 28553 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41979
I20260810 07:24:41.200904 28553 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:41.201145 28553 heartbeater.cc:511] Master 127.24.60.254:41979 requested a full tablet report, sending...
I20260810 07:24:41.201953 28330 ts_manager.cc:194] Registered new tserver with Master: e7a25bb9568f4fb3878fd6ea89593a1e (127.24.60.193:39921)
I20260810 07:24:41.203001 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:40003
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:41.224850 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41979
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:41.312222 28559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:41.312470 28559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:41.312498 28559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:41.313794 28559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:41.313844 28559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:41.313874 28559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:41.315397 28559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:41.315626 28559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:41.315820 28559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:41.321787 28565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:41.321681 28564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:41.321676 28559 server_base.cc:1034] running on GCE node
W20260810 07:24:41.321692 28567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:41.322182 28559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:41.322435 28559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:41.323616 28559 hybrid_clock.cc:648] HybridClock initialized: now 1786346681323597 us; error 31 us; skew 500 ppm
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:41.326422 28559 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:41.326956 28559 webserver.cc:460] Webserver started at http://127.24.60.194:44731/ using document root <none> and password file <none>
I20260810 07:24:41.327225 28559 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:41.327298 28559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:41.327483 28559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:41.328637 28559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "f67b95e2e5144fea855c000f4b5450bf"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "f2b2d0e2007cdb49825e55a7bdbdfe11"
server_key_iv: "b9ba927f264e2976f7075b0d2880e5e6"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.329023 28559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "f67b95e2e5144fea855c000f4b5450bf"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "f2b2d0e2007cdb49825e55a7bdbdfe11"
server_key_iv: "b9ba927f264e2976f7075b0d2880e5e6"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.330627 28559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:41.331489 28574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.331727 28559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:41.331809 28559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "f67b95e2e5144fea855c000f4b5450bf"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "f2b2d0e2007cdb49825e55a7bdbdfe11"
server_key_iv: "b9ba927f264e2976f7075b0d2880e5e6"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.331905 28559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:41.350455 28559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:41.352950 28559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:41.353133 28559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:41.353507 28559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:41.353859 28559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:41.353919 28559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.353953 28559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:41.354004 28559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.360522 28559 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:41567
I20260810 07:24:41.360615 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:41567 every 8 connection(s)
I20260810 07:24:41.360981 28559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:41.361219 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28559
I20260810 07:24:41.361351 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:41.361627 24819 external_mini_cluster.cc:1428] Setting key d898fac82a56f163a8747f8d9797d43b
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
I20260810 07:24:41.373672 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.362429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:53083 (local address 127.24.60.254:41979)
0810 07:24:41.362516 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:41.362519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:41.362530 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:41.363204 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:41.363206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:41.363224 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:41.363289 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:41.364601 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.365302 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.366610 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.366950 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.367051 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.369988 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:41.369990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:41.369991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:41.370000 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:41.371875 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.372466 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.372468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.372468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.372541 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.372841 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.372842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.372843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.372986 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:41.373067 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:41.373460 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 07:24:41.373573 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:24:41.373893 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.362115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41979 (local address 127.24.60.194:53083)
0810 07:24:41.362302 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:41.362365 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:41.363085 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:41.363387 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:41.363391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:41.364155 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:41.364492 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:41.364501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.365415 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.365417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:41.366462 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:41.366465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.367061 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.367063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:41.367232 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.367892 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:41.367903 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:41.369853 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:41.371985 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:41.371996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:41.371999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:41.372350 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:41.372621 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:41.372622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:41.372622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:41.372755 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:41.373058 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:41.373067 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:41.373297 (+ 230us) client_negotiation.cc:769] Sending connection context
0810 07:24:41.373434 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 07:24:41.373625 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
I20260810 07:24:41.374615 28709 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41979
I20260810 07:24:41.374727 28709 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:41.374920 28709 heartbeater.cc:511] Master 127.24.60.254:41979 requested a full tablet report, sending...
I20260810 07:24:41.375432 28330 ts_manager.cc:194] Registered new tserver with Master: f67b95e2e5144fea855c000f4b5450bf (127.24.60.194:41567)
I20260810 07:24:41.375833 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:53083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:41.393744 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41979
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:41.480443 28714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:41.480726 28714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:41.480769 28714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:41.482101 28714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:41.482179 28714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:41.482228 28714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:41.483853 28714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:41.484156 28714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:41.484416 28714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:41.489938 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:41.489938 28724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:41.490162 28721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:41.490612 28714 server_base.cc:1034] running on GCE node
I20260810 07:24:41.490854 28714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:41.491143 28714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:41.492334 28714 hybrid_clock.cc:648] HybridClock initialized: now 1786346681492307 us; error 44 us; skew 500 ppm
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:41.495887 28714 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:41.496551 28714 webserver.cc:460] Webserver started at http://127.24.60.195:37829/ using document root <none> and password file <none>
I20260810 07:24:41.496837 28714 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:41.496889 28714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:41.496995 28714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:41.498250 28714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "52ab9448b21648489a83faa37413d5de"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "4a823a41aca90a505b77327d943b663a"
server_key_iv: "eec3454738d78420751e95173c56a82e"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.498589 28714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "52ab9448b21648489a83faa37413d5de"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "4a823a41aca90a505b77327d943b663a"
server_key_iv: "eec3454738d78420751e95173c56a82e"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.500089 28714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:41.500998 28731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.501257 28714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:41.501374 28714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "52ab9448b21648489a83faa37413d5de"
format_stamp: "Formatted at 2026-08-10 07:24:41 on dist-test-slave-vxj2"
server_key: "4a823a41aca90a505b77327d943b663a"
server_key_iv: "eec3454738d78420751e95173c56a82e"
server_key_version: "encryptionkey@0"
I20260810 07:24:41.501492 28714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:41.514349 28714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:41.516855 28714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:41.517024 28714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:41.517306 28714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:41.517671 28714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:41.517728 28714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.517772 28714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:41.517803 28714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:41.524055 28714 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:33047
I20260810 07:24:41.524152 28860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:33047 every 8 connection(s)
I20260810 07:24:41.524538 28714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:41.530339 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28714
I20260810 07:24:41.530463 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:41.530639 24819 external_mini_cluster.cc:1428] Setting key 60a8106b8683207a715d1857be114c10
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:41.535697 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.525532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:44723 (local address 127.24.60.254:41979)
0810 07:24:41.525606 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:41.525609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:41.525855 (+ 246us) server_negotiation.cc:407] Connection header received
0810 07:24:41.526395 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:41.526396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:41.526409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:41.526470 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:41.527504 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.528045 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.529134 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.529420 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.529488 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.532527 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:41.532528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:41.532529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:41.532537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:41.534098 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.534611 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.534612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.534613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.534670 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.535051 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.535052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.535052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.535175 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:41.535234 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:41.535553 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 07:24:41.535627 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:24:41.536070 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.525506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41979 (local address 127.24.60.195:44723)
0810 07:24:41.525711 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:41.525752 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:41.526269 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:41.526548 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:41.526552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:41.527173 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:41.527399 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:41.527405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.528154 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.528155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:41.529038 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:41.529041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.529537 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.529538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:41.529669 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.530400 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:41.530411 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:41.532389 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:41.534195 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:41.534205 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:41.534208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:41.534535 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:41.534752 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:41.534753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:41.534753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:41.534962 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:41.535243 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:41.535249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:41.535424 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 07:24:41.535527 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 07:24:41.535683 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":113,"threads_started":1}
I20260810 07:24:41.536721 28861 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41979
I20260810 07:24:41.536924 28861 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:41.537200 28861 heartbeater.cc:511] Master 127.24.60.254:41979 requested a full tablet report, sending...
I20260810 07:24:41.537804 28330 ts_manager.cc:194] Registered new tserver with Master: 52ab9448b21648489a83faa37413d5de (127.24.60.195:33047)
I20260810 07:24:41.538219 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:44723
I20260810 07:24:41.542730 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346681, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:41.568739 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:41.560152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.24.60.254:41979)
0810 07:24:41.560231 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:41.560233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:41.560241 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:41.560265 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:41.560266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:41.560275 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:41.560345 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:41.561019 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.561648 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.562448 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:41.562667 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:41.562751 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:41.565594 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:41.565596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:41.565596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:41.565613 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:41.567294 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.567780 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.567781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.567782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.567860 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:41.568076 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:41.568077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:41.568078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:41.568266 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:41.568394 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:41.568527 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 07:24:41.568627 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 07:24:41.569145 28330 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51444
I20260810 07:24:41.569754 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28401
I20260810 07:24:41.582013 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28559
I20260810 07:24:41.590888 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28714
I20260810 07:24:41.600132 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28292
2026-08-10T07:24:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2063 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28877](info): set up 2 sockets
Aug 10 07:24:41 dist-test-slave-vxj2 krb5kdc[28877](info): commencing operation
krb5kdc: starting...
W20260810 07:24:42.736847 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:42 dist-test-slave-vxj2 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:42.873027 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:39817
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:42.956883 28892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:42.957116 28892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:42.957139 28892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:42.958391 28892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:42.958485 28892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:42.958513 28892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:42.958541 28892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:42.958573 28892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:42.960244 28892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:42.960552 28892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:42.960796 28892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:42.966143 28904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:42.966126 28901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:42.966126 28906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:42.966459 28892 server_base.cc:1034] running on GCE node
I20260810 07:24:42.966795 28892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:42.967125 28892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:42.968300 28892 hybrid_clock.cc:648] HybridClock initialized: now 1786346682968270 us; error 49 us; skew 500 ppm
Aug 10 07:24:42 dist-test-slave-vxj2 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346682, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:42.982247 28892 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:42.982841 28892 webserver.cc:460] Webserver started at http://127.24.60.254:41599/ using document root <none> and password file <none>
I20260810 07:24:42.983103 28892 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:42.983176 28892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:42.983335 28892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:42.984504 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "0ac290cb1cdc4f5da97129c26a416cd8"
format_stamp: "Formatted at 2026-08-10 07:24:42 on dist-test-slave-vxj2"
server_key: "e483074a8d2e79e915724477a4e5a6a4"
server_key_iv: "07ca81f5864e3537a2b8d949bf4e3e46"
server_key_version: "encryptionkey@0"
I20260810 07:24:42.984895 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "0ac290cb1cdc4f5da97129c26a416cd8"
format_stamp: "Formatted at 2026-08-10 07:24:42 on dist-test-slave-vxj2"
server_key: "e483074a8d2e79e915724477a4e5a6a4"
server_key_iv: "07ca81f5864e3537a2b8d949bf4e3e46"
server_key_version: "encryptionkey@0"
I20260810 07:24:42.986421 28892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:24:42.987329 28913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:42.987534 28892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:42.987650 28892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "0ac290cb1cdc4f5da97129c26a416cd8"
format_stamp: "Formatted at 2026-08-10 07:24:42 on dist-test-slave-vxj2"
server_key: "e483074a8d2e79e915724477a4e5a6a4"
server_key_iv: "07ca81f5864e3537a2b8d949bf4e3e46"
server_key_version: "encryptionkey@0"
I20260810 07:24:42.987728 28892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:43.010344 28892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:43.013027 28892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:43.013350 28892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:43.017987 28892 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:39817
I20260810 07:24:43.018052 28969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:39817 every 8 connection(s)
I20260810 07:24:43.018683 28892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:43.019207 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28892
I20260810 07:24:43.019351 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:43.019569 24819 external_mini_cluster.cc:1428] Setting key cea92d60a70453c33f586e5d8ecf8c8e
I20260810 07:24:43.019644 28970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:43.023433 28970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8: Bootstrap starting.
I20260810 07:24:43.024260 28970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:43.024605 28970 log.cc:824] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:43.025542 28970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8: No bootstrap required, opened a new log
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:43.028165 28970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 39817 } }
I20260810 07:24:43.028375 28970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:43.028445 28970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac290cb1cdc4f5da97129c26a416cd8, State: Initialized, Role: FOLLOWER
I20260810 07:24:43.028615 28970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 39817 } }
I20260810 07:24:43.028740 28970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:43.028781 28970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:43.028829 28970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:43.029723 28970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 39817 } }
I20260810 07:24:43.029847 28970 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac290cb1cdc4f5da97129c26a416cd8; no voters:
I20260810 07:24:43.030077 28970 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:43.030118 28977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:43.030316 28977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac290cb1cdc4f5da97129c26a416cd8, State: Running, Role: LEADER
I20260810 07:24:43.030467 28970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:43.030490 28977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 39817 } }
I20260810 07:24:43.030959 28977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac290cb1cdc4f5da97129c26a416cd8. Latest consensus state: current_term: 1 leader_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 39817 } } }
I20260810 07:24:43.031147 28977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:43.031440 28986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:43.030968 28978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac290cb1cdc4f5da97129c26a416cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 39817 } } }
I20260810 07:24:43.031950 28978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:43.032081 28986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:43.032644 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.020170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.24.60.254:39817)
0810 07:24:43.020390 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:43.020398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:43.020432 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:24:43.020891 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:43.020900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:43.020952 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:43.021080 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:43.022018 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.023053 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.024191 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.024736 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.024851 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.027739 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:43.027750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:43.027751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:43.027764 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:43.030097 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.030629 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.030632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.030632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.030801 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.031074 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.031075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.031076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.031436 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:43.031596 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:43.032228 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 07:24:43.032402 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260810 07:24:43.034328 28986 catalog_manager.cc:1347] Generated new cluster ID: 04569826823a4666bc16a3f9de8d206e
I20260810 07:24:43.034396 28986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:43.053146 28986 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:43.053962 28986 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:43.066995 28986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac290cb1cdc4f5da97129c26a416cd8: Generated new TSK 0
I20260810 07:24:43.067245 28986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:43.104053 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:39817
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:43.194636 28999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:43.194890 28999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:43.194918 28999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:43.196247 28999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:43.196307 28999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:43.196338 28999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:43.198047 28999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:39817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:43.198273 28999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:43.198487 28999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:43.204035 29007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:43.204054 29004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:43.204061 29005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:43.204274 28999 server_base.cc:1034] running on GCE node
I20260810 07:24:43.204533 28999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:43.204850 28999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:43.206023 28999 hybrid_clock.cc:648] HybridClock initialized: now 1786346683205994 us; error 44 us; skew 500 ppm
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:43.208731 28999 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:43.209364 28999 webserver.cc:460] Webserver started at http://127.24.60.193:37137/ using document root <none> and password file <none>
I20260810 07:24:43.209635 28999 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:43.209707 28999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:43.209870 28999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:43.211014 28999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "b838bede60b34380b1eef29f88a34eb0"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "ed3118056ac88b5a57ec32cf00f03570"
server_key_iv: "8a43caba73f98e87910b0eebc711b950"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.211405 28999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "b838bede60b34380b1eef29f88a34eb0"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "ed3118056ac88b5a57ec32cf00f03570"
server_key_iv: "8a43caba73f98e87910b0eebc711b950"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.212996 28999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:43.214079 29015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.214365 28999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:43.214502 28999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "b838bede60b34380b1eef29f88a34eb0"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "ed3118056ac88b5a57ec32cf00f03570"
server_key_iv: "8a43caba73f98e87910b0eebc711b950"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.214598 28999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:43.235808 28999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:43.238157 28999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:43.238348 28999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:43.238636 28999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:43.238991 28999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:43.239049 28999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.239087 28999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:43.239135 28999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.245376 28999 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:33511
I20260810 07:24:43.245448 29148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:33511 every 8 connection(s)
I20260810 07:24:43.245779 28999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:43.250257 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 28999
I20260810 07:24:43.250370 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:43.250701 24819 external_mini_cluster.cc:1428] Setting key c71b322f40e2a1707dc618e52ada1f5a
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:43.256940 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.246604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:59589 (local address 127.24.60.254:39817)
0810 07:24:43.246697 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:43.246700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:43.246885 (+ 185us) server_negotiation.cc:407] Connection header received
0810 07:24:43.247957 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:43.247963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:43.247982 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:43.248055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:43.248795 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.249335 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.250547 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.250886 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.250965 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.253612 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:43.253614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:43.253614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:43.253625 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:43.255328 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.255771 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.255773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.255774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.255847 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.256145 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.256146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.256146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.256308 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:43.256386 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:43.256683 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 07:24:43.256792 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:24:43.256968 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.246546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:39817 (local address 127.24.60.193:59589)
0810 07:24:43.246748 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:43.246787 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:43.247796 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:43.248142 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:43.248146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:43.248444 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:43.248678 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:43.248683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.249429 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.249429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:43.250309 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:43.250311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.250980 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.250982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:43.251127 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.251728 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:43.251738 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:43.253497 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:43.255423 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:43.255431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:43.255433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:43.255683 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:43.255930 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:43.255931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:43.255932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:43.256070 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:43.256395 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:43.256401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:43.256537 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 07:24:43.256628 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 07:24:43.256765 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260810 07:24:43.257992 29149 heartbeater.cc:347] Connected to a master server at 127.24.60.254:39817
I20260810 07:24:43.258119 29149 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:43.258348 29149 heartbeater.cc:511] Master 127.24.60.254:39817 requested a full tablet report, sending...
I20260810 07:24:43.259018 28933 ts_manager.cc:194] Registered new tserver with Master: b838bede60b34380b1eef29f88a34eb0 (127.24.60.193:33511)
I20260810 07:24:43.259678 28933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:59589
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:43.281772 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:39817
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:43.368423 29154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:43.368705 29154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:43.368753 29154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:43.370199 29154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:43.370292 29154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:43.370342 29154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:43.372226 29154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:39817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:43.372565 29154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:43.372833 29154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:43.378528 29161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:43.378587 29162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:43.378528 29164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:43.378721 29154 server_base.cc:1034] running on GCE node
I20260810 07:24:43.379058 29154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:43.379316 29154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:43.380573 29154 hybrid_clock.cc:648] HybridClock initialized: now 1786346683380553 us; error 52 us; skew 500 ppm
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:43.385041 29154 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:43.385887 29154 webserver.cc:460] Webserver started at http://127.24.60.194:44613/ using document root <none> and password file <none>
I20260810 07:24:43.386264 29154 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:43.386358 29154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:43.386541 29154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:43.388075 29154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "5887449d39694fdea08387c5a2adbbf4"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "164366e6cc3ac62bbab73fe580f797ca"
server_key_iv: "14a791bd6a6ce648982e980a4d9433e7"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.388509 29154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "5887449d39694fdea08387c5a2adbbf4"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "164366e6cc3ac62bbab73fe580f797ca"
server_key_iv: "14a791bd6a6ce648982e980a4d9433e7"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.390316 29154 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 07:24:43.391300 29172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.391547 29154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:43.391685 29154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "5887449d39694fdea08387c5a2adbbf4"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "164366e6cc3ac62bbab73fe580f797ca"
server_key_iv: "14a791bd6a6ce648982e980a4d9433e7"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.391798 29154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:43.420744 29154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:43.423403 29154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:43.423597 29154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:43.423882 29154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:43.424247 29154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:43.424316 29154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.424362 29154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:43.424397 29154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.431734 29154 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:42779
I20260810 07:24:43.431782 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:42779 every 8 connection(s)
I20260810 07:24:43.432220 29154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:43.438268 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 29154
I20260810 07:24:43.438395 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:43.438591 24819 external_mini_cluster.cc:1428] Setting key 3c694ccce610ec01909d15cfaaddbde0
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:43.443642 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.433114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:42535 (local address 127.24.60.254:39817)
0810 07:24:43.433207 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:43.433210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:43.433469 (+ 259us) server_negotiation.cc:407] Connection header received
0810 07:24:43.434500 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:43.434502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:43.434514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:43.434585 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:43.435204 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.435747 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.436798 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.437052 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.437118 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.440071 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:43.440073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:43.440074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:43.440082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:43.442032 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.442498 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.442499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.442500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.442578 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.442840 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.442841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.442842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.443017 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:43.443100 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:43.443366 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 07:24:43.443492 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:24:43.443634 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.433093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:39817 (local address 127.24.60.194:42535)
0810 07:24:43.433322 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:43.433364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:43.434381 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:43.434643 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:43.434646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:43.434927 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:43.435129 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:43.435133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.435833 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.435834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:43.436714 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:43.436717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.437220 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.437221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:43.437345 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.437948 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:43.437957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:43.439928 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:43.442143 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:43.442151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:43.442154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:43.442406 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:43.442657 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:43.442658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:43.442658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:43.442770 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:43.443090 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:43.443095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:43.443221 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 07:24:43.443307 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:24:43.443444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":149,"threads_started":1}
I20260810 07:24:43.444504 29301 heartbeater.cc:347] Connected to a master server at 127.24.60.254:39817
I20260810 07:24:43.444639 29301 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:43.444903 29301 heartbeater.cc:511] Master 127.24.60.254:39817 requested a full tablet report, sending...
I20260810 07:24:43.445350 28933 ts_manager.cc:194] Registered new tserver with Master: 5887449d39694fdea08387c5a2adbbf4 (127.24.60.194:42779)
I20260810 07:24:43.445767 28933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:42535
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:43.468945 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:39817
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:43.556456 29306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:43.556702 29306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:43.556731 29306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:43.558033 29306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:43.558091 29306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:43.558121 29306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:43.559717 29306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:39817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:43.560010 29306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:43.560226 29306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:43.565900 29314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:43.566075 29312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:43.565900 29311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:43.566756 29306 server_base.cc:1034] running on GCE node
I20260810 07:24:43.567106 29306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:43.567385 29306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:43.568557 29306 hybrid_clock.cc:648] HybridClock initialized: now 1786346683568538 us; error 37 us; skew 500 ppm
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:43.571893 29306 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:43.572494 29306 webserver.cc:460] Webserver started at http://127.24.60.195:39333/ using document root <none> and password file <none>
I20260810 07:24:43.572748 29306 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:43.572821 29306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:43.572983 29306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:43.574294 29306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "e593dec66a7f48c0905784b6b8f3509e"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "9e80f3c79b8170902c2f421df0baca33"
server_key_iv: "990f14c3f0f3dc842b3887a011b11fc0"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.574779 29306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "e593dec66a7f48c0905784b6b8f3509e"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "9e80f3c79b8170902c2f421df0baca33"
server_key_iv: "990f14c3f0f3dc842b3887a011b11fc0"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.576346 29306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:24:43.577206 29321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.577430 29306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:43.577525 29306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "e593dec66a7f48c0905784b6b8f3509e"
format_stamp: "Formatted at 2026-08-10 07:24:43 on dist-test-slave-vxj2"
server_key: "9e80f3c79b8170902c2f421df0baca33"
server_key_iv: "990f14c3f0f3dc842b3887a011b11fc0"
server_key_version: "encryptionkey@0"
I20260810 07:24:43.577589 29306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:43.601418 29306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:43.603772 29306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:43.603906 29306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:43.604133 29306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:43.604498 29306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:43.604534 29306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.604558 29306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:43.604609 29306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:43.611181 29306 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:45419
I20260810 07:24:43.611335 29451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:45419 every 8 connection(s)
I20260810 07:24:43.611626 29306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:43.615165 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 29306
I20260810 07:24:43.615273 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:43.615465 24819 external_mini_cluster.cc:1428] Setting key b4aad9edb1ab5aba06056837da90e019
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:43.622913 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.612485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:60109 (local address 127.24.60.254:39817)
0810 07:24:43.612712 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:43.612731 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:43.612915 (+ 184us) server_negotiation.cc:407] Connection header received
0810 07:24:43.614076 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:43.614078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:43.614090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:43.614147 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:43.614861 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.615647 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.616875 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.617097 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.617192 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.619769 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:43.619771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:43.619772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:43.619781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:43.621379 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.621845 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.621846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.621847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.621912 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.622197 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.622199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.622199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.622327 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:43.622407 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:43.622722 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 07:24:43.622813 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260810 07:24:43.623167 29454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.612448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:39817 (local address 127.24.60.195:60109)
0810 07:24:43.612747 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:43.612810 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:43.613935 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:43.614235 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:43.614238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:43.614540 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:43.614750 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:43.614754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.615774 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.615777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:43.616774 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:43.616778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.617222 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.617224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:43.617364 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.618035 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:43.618044 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:43.619647 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:43.621476 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:43.621487 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:43.621490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:43.621757 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:43.621988 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:43.621989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:43.621990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:43.622123 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:43.622410 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:43.622419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:43.622583 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 07:24:43.622699 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 07:24:43.622882 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260810 07:24:43.623729 29452 heartbeater.cc:347] Connected to a master server at 127.24.60.254:39817
I20260810 07:24:43.623849 29452 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:43.624058 29452 heartbeater.cc:511] Master 127.24.60.254:39817 requested a full tablet report, sending...
I20260810 07:24:43.624477 28933 ts_manager.cc:194] Registered new tserver with Master: e593dec66a7f48c0905784b6b8f3509e (127.24.60.195:45419)
I20260810 07:24:43.624884 28933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:60109
I20260810 07:24:43.627458 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:43.652971 29461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.644458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.24.60.193:33511)
0810 07:24:43.644639 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:43.644644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:43.644662 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:24:43.644692 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:43.644694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:43.644708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:43.644791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:43.645518 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.646270 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.647074 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.647425 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.647491 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.649988 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:43.649999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:43.650000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:43.650008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:43.651682 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.652083 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.652087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.652087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.652151 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.652364 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.652365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.652365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.652531 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:43.652706 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:43.652751 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 07:24:43.652838 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[28877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:43.662572 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:43.654485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.24.60.254:39817)
0810 07:24:43.654528 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:43.654531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:43.654614 (+ 83us) server_negotiation.cc:407] Connection header received
0810 07:24:43.654690 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:43.654690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:43.654700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:43.654748 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:43.655315 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.655866 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.656660 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:43.656901 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:43.656968 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:43.659434 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:43.659436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:43.659436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:43.659445 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:43.661145 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.661510 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.661511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.661511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.661571 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:43.662034 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:43.662035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:43.662035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:43.662211 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:43.662320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:43.662431 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 07:24:43.662502 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 07:24:43.662876 28933 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44796
I20260810 07:24:43.663335 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28999
I20260810 07:24:43.673856 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 29154
I20260810 07:24:43.681521 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 29306
I20260810 07:24:43.690639 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 28892
2026-08-10T07:24:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2087 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[29466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[29466](info): set up 2 sockets
Aug 10 07:24:43 dist-test-slave-vxj2 krb5kdc[29466](info): commencing operation
krb5kdc: starting...
W20260810 07:24:44.851075 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:44 dist-test-slave-vxj2 krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:44.984750 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34043
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:34043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:45.070597 29479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:45.070833 29479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:45.070860 29479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:45.072122 29479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:45.072175 29479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:45.072189 29479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:45.072202 29479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:45.072248 29479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:45.073964 29479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:34043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:45.074261 29479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:45.074535 29479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:45.079903 29489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:45.080029 29479 server_base.cc:1034] running on GCE node
W20260810 07:24:45.079885 29487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.079859 29486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:45.080508 29479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:45.080850 29479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:45.082028 29479 hybrid_clock.cc:648] HybridClock initialized: now 1786346685082019 us; error 57 us; skew 500 ppm
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:45.084970 29479 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:45.085469 29479 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:45.085548 29479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:45.085716 29479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:45.086881 29479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "bab123c717994394b42ccf9ec2afefbe"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "095d395df872317acc159d8ab77abb52"
server_key_iv: "ebeb77e223776b45904653b4998ec732"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.087273 29479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "bab123c717994394b42ccf9ec2afefbe"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "095d395df872317acc159d8ab77abb52"
server_key_iv: "ebeb77e223776b45904653b4998ec732"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.088723 29479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:45.089720 29497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.089990 29479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:24:45.090107 29479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "bab123c717994394b42ccf9ec2afefbe"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "095d395df872317acc159d8ab77abb52"
server_key_iv: "ebeb77e223776b45904653b4998ec732"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.090191 29479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:45.115284 29479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:45.117936 29479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:45.118126 29479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:45.122574 29479 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:34043
I20260810 07:24:45.122609 29566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:34043 every 8 connection(s)
I20260810 07:24:45.122925 29479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:45.123993 29567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:45.126633 29567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe: Bootstrap starting.
I20260810 07:24:45.127292 29567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:45.127630 29567 log.cc:824] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:45.128432 29567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe: No bootstrap required, opened a new log
I20260810 07:24:45.130246 29567 raft_consensus.cc:348] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab123c717994394b42ccf9ec2afefbe" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34043 } }
I20260810 07:24:45.130411 29567 raft_consensus.cc:374] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:45.130436 29567 raft_consensus.cc:729] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab123c717994394b42ccf9ec2afefbe, State: Initialized, Role: FOLLOWER
I20260810 07:24:45.130607 29567 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab123c717994394b42ccf9ec2afefbe" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34043 } }
I20260810 07:24:45.130692 29567 raft_consensus.cc:388] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:45.130738 29567 raft_consensus.cc:482] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:45.130800 29567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:45.130978 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 29479
I20260810 07:24:45.131099 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:45.131304 24819 external_mini_cluster.cc:1428] Setting key 23771377d2581b50e63fb7a09d509178
I20260810 07:24:45.132025 29567 raft_consensus.cc:504] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab123c717994394b42ccf9ec2afefbe" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34043 } }
I20260810 07:24:45.132202 29567 leader_election.cc:302] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bab123c717994394b42ccf9ec2afefbe; no voters:
I20260810 07:24:45.132407 29567 leader_election.cc:288] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:45.132539 29572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:45.132686 29567 sys_catalog.cc:564] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:45.132742 29572 raft_consensus.cc:686] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [term 1 LEADER]: Becoming Leader. State: Replica: bab123c717994394b42ccf9ec2afefbe, State: Running, Role: LEADER
I20260810 07:24:45.132895 29572 consensus_queue.cc:237] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab123c717994394b42ccf9ec2afefbe" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34043 } }
I20260810 07:24:45.133540 29572 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bab123c717994394b42ccf9ec2afefbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab123c717994394b42ccf9ec2afefbe" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34043 } } }
I20260810 07:24:45.133656 29572 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:45.133848 29572 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab123c717994394b42ccf9ec2afefbe. Latest consensus state: current_term: 1 leader_uuid: "bab123c717994394b42ccf9ec2afefbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab123c717994394b42ccf9ec2afefbe" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34043 } } }
I20260810 07:24:45.133929 29572 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:45.134091 29584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:45.135339 29584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:45.137687 29584 catalog_manager.cc:1347] Generated new cluster ID: accdff019e6e4ca1b4be4231803ed1fa
I20260810 07:24:45.137820 29584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:45.145630 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.131972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.24.60.254:34043)
0810 07:24:45.132176 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.132185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.132214 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:24:45.132347 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.132355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.132419 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.132592 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:45.133643 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.135246 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.136342 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.137010 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.137211 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.139971 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:45.139976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:45.139976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:45.139991 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:45.143435 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.144065 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.144069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.144070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.144235 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.144485 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.144486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.144486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.144884 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:45.145077 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.145198 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.145371 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
I20260810 07:24:45.154403 29584 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:45.155057 29584 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:45.162628 29584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe: Generated new TSK 0
I20260810 07:24:45.162962 29584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:45.193043 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34043
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:45.281592 29597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:45.281833 29597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:45.281862 29597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:45.283182 29597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:45.283239 29597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:45.283270 29597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:45.284991 29597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:45.285315 29597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:45.285588 29597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:45.291344 29607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.291335 29605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.291329 29603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:45.291425 29597 server_base.cc:1034] running on GCE node
I20260810 07:24:45.291802 29597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:45.292083 29597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:45.293217 29597 hybrid_clock.cc:648] HybridClock initialized: now 1786346685293204 us; error 31 us; skew 500 ppm
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:45.296018 29597 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:45.296403 29597 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:45.296478 29597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:45.296638 29597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:45.297816 29597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "69fffd1e464746b59dc825b364b24562"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "c96139f2ba4e8fd64997b418c87edec1"
server_key_iv: "00e7e1487cd4a732d4a9c3aae2f0f68f"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.298183 29597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "69fffd1e464746b59dc825b364b24562"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "c96139f2ba4e8fd64997b418c87edec1"
server_key_iv: "00e7e1487cd4a732d4a9c3aae2f0f68f"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.299652 29597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:45.300601 29613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.300843 29597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:45.300966 29597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "69fffd1e464746b59dc825b364b24562"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "c96139f2ba4e8fd64997b418c87edec1"
server_key_iv: "00e7e1487cd4a732d4a9c3aae2f0f68f"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.301049 29597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:45.318933 29597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:45.321419 29597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:45.321593 29597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:45.321849 29597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:45.322204 29597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:45.322264 29597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.322309 29597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:45.322333 29597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.328560 29597 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:43311
I20260810 07:24:45.328598 29745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:43311 every 8 connection(s)
I20260810 07:24:45.328948 29597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:45.329880 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 29597
I20260810 07:24:45.330018 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:45.330247 24819 external_mini_cluster.cc:1428] Setting key e34b13d89064a5fc63bd9e32e254f4eb
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
I20260810 07:24:45.342011 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.329837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:40249 (local address 127.24.60.254:34043)
0810 07:24:45.329915 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.329918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.330118 (+ 200us) server_negotiation.cc:407] Connection header received
0810 07:24:45.330938 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.330939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.330956 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.331045 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:45.332187 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.333012 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.334969 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.335206 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.335303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.338431 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:45.338434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:45.338434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:45.338445 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:45.340149 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.340719 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.340721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.340722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.340794 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.341130 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.341131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.341132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.341311 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:45.341391 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.341791 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.341931 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:24:45.342196 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.329785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34043 (local address 127.24.60.193:40249)
0810 07:24:45.329977 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.330025 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.330801 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.331152 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.331157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.331492 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:45.332061 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.332067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.333118 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.333119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.334817 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.334821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.335396 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.335398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.335593 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.336256 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:45.336270 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:45.338290 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:45.340288 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:45.340301 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:45.340305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:45.340630 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:45.340930 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:45.340931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:45.340931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:45.341054 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:45.341395 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:45.341405 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:45.341595 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.341762 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.341952 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260810 07:24:45.342836 29746 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34043
I20260810 07:24:45.342947 29746 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:45.343132 29746 heartbeater.cc:511] Master 127.24.60.254:34043 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
I20260810 07:24:45.343832 29521 ts_manager.cc:194] Registered new tserver with Master: 69fffd1e464746b59dc825b364b24562 (127.24.60.193:43311)
I20260810 07:24:45.344560 29521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:40249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:45.362234 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34043
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:45.454777 29751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:45.454995 29751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:45.455022 29751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:45.456248 29751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:45.456300 29751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:45.456328 29751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:45.457983 29751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:45.458262 29751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:45.458542 29751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:45.463958 29758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.463958 29760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.463979 29757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:45.464260 29751 server_base.cc:1034] running on GCE node
I20260810 07:24:45.464581 29751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:45.464845 29751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:45.466035 29751 hybrid_clock.cc:648] HybridClock initialized: now 1786346685466017 us; error 53 us; skew 500 ppm
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:45.469079 29751 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:45.469503 29751 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:45.469584 29751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:45.469743 29751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:45.470896 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "f75059f08ca041f7acc90ad43a68bb9e"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "60e22620f88d1425d8a027a1982afc76"
server_key_iv: "5cf2b83db8488a72a84d02cd46f7a3e1"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.471283 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "f75059f08ca041f7acc90ad43a68bb9e"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "60e22620f88d1425d8a027a1982afc76"
server_key_iv: "5cf2b83db8488a72a84d02cd46f7a3e1"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.472689 29751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:45.473698 29766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.473905 29751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:24:45.474023 29751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "f75059f08ca041f7acc90ad43a68bb9e"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "60e22620f88d1425d8a027a1982afc76"
server_key_iv: "5cf2b83db8488a72a84d02cd46f7a3e1"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.474110 29751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:45.491029 29751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:45.493327 29751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:45.493554 29751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:45.493837 29751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:45.494197 29751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:45.494256 29751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.494292 29751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:45.494324 29751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.500823 29751 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:40175
I20260810 07:24:45.500901 29901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:40175 every 8 connection(s)
I20260810 07:24:45.501262 29751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:45.508687 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 29751
I20260810 07:24:45.508814 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:45.509045 24819 external_mini_cluster.cc:1428] Setting key 4ac80c0ad2a73e0ff28a0d8bb200d65c
I20260810 07:24:45.512367 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.502135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:47599 (local address 127.24.60.254:34043)
0810 07:24:45.502187 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.502189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.502394 (+ 205us) server_negotiation.cc:407] Connection header received
0810 07:24:45.502878 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.502879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.502889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.502954 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:45.503992 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.504523 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.505724 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.505992 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.506072 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.508616 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:45.508618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:45.508619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:45.508630 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:45.510584 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.511153 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.511155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.511155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.511235 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.511564 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.511565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.511565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.511721 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:45.511821 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.512152 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.512259 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:24:45.512657 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.502054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34043 (local address 127.24.60.194:47599)
0810 07:24:45.502235 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.502280 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.502778 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.503009 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.503012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.503294 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:45.503815 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.503819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.504596 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.504597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.505628 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.505631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.506107 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.506108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.506260 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.506824 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:45.506833 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:45.508481 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:45.510704 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:45.510718 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:45.510721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:45.511024 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:45.511331 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:45.511334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:45.511335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:45.511476 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:45.511805 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:45.511814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:45.511985 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.512115 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.512332 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260810 07:24:45.513254 29902 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34043
I20260810 07:24:45.513383 29902 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:45.513578 29902 heartbeater.cc:511] Master 127.24.60.254:34043 requested a full tablet report, sending...
I20260810 07:24:45.514135 29521 ts_manager.cc:194] Registered new tserver with Master: f75059f08ca041f7acc90ad43a68bb9e (127.24.60.194:40175)
I20260810 07:24:45.514776 29521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:47599
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:45.541586 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34043
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:45.631376 29907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:45.631664 29907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:45.631714 29907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:45.633061 29907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:45.633142 29907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:45.633229 29907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:45.635185 29907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:45.635519 29907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:45.635811 29907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:45.641415 29913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.641415 29914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:45.641415 29918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:45.641628 29907 server_base.cc:1034] running on GCE node
I20260810 07:24:45.641912 29907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:45.642144 29907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:45.643292 29907 hybrid_clock.cc:648] HybridClock initialized: now 1786346685643281 us; error 29 us; skew 500 ppm
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:45.646196 29907 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:45.646647 29907 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:45.646700 29907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:45.646840 29907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:45.648046 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "3f79d3b539b347f382d50c7eadb5fac4"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "0560662c0239db0c144cdb81b51f3cb3"
server_key_iv: "4d085b554412506e6ba4c635379d6bd3"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.648581 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "3f79d3b539b347f382d50c7eadb5fac4"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "0560662c0239db0c144cdb81b51f3cb3"
server_key_iv: "4d085b554412506e6ba4c635379d6bd3"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.650159 29907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:45.651006 29924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.651219 29907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:45.651314 29907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "3f79d3b539b347f382d50c7eadb5fac4"
format_stamp: "Formatted at 2026-08-10 07:24:45 on dist-test-slave-vxj2"
server_key: "0560662c0239db0c144cdb81b51f3cb3"
server_key_iv: "4d085b554412506e6ba4c635379d6bd3"
server_key_version: "encryptionkey@0"
I20260810 07:24:45.651377 29907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:45.676050 29907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:45.679031 29907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:45.679171 29907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:45.679391 29907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:45.679780 29907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:45.679924 29907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.679951 29907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:45.679963 29907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:45.686221 29907 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:36339
I20260810 07:24:45.686296 30055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:36339 every 8 connection(s)
I20260810 07:24:45.686616 29907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:45.687930 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 29907
I20260810 07:24:45.688033 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:45.688215 24819 external_mini_cluster.cc:1428] Setting key 2f4a4c062813f1263e66f1ab9f351699
Aug 10 07:24:45 dist-test-slave-vxj2 krb5kdc[29466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:45.697286 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.687414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:50795 (local address 127.24.60.254:34043)
0810 07:24:45.687499 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.687501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.687706 (+ 205us) server_negotiation.cc:407] Connection header received
0810 07:24:45.688449 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.688452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.688467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.688550 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:45.689822 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.690324 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.691364 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.691557 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.691620 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.694218 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:45.694220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:45.694220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:45.694228 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:45.695752 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.696232 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.696233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.696233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.696290 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.696548 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.696548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.696549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.696667 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:45.696721 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.697070 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.697140 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:24:45.697539 30058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.687369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34043 (local address 127.24.60.195:50795)
0810 07:24:45.687563 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.687609 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.688305 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.688649 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.688654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.689062 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:45.689699 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.689706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.690422 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.690423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.691256 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.691258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.691646 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.691647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.691760 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.692381 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:45.692389 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:45.694106 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:45.695864 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:45.695872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:45.695881 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:45.696154 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:45.696357 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:45.696357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:45.696358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:45.696481 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:45.696728 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:45.696733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:45.696863 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.697043 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.697274 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260810 07:24:45.698058 30056 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34043
I20260810 07:24:45.698189 30056 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:45.698360 30056 heartbeater.cc:511] Master 127.24.60.254:34043 requested a full tablet report, sending...
I20260810 07:24:45.698829 29521 ts_manager.cc:194] Registered new tserver with Master: 3f79d3b539b347f382d50c7eadb5fac4 (127.24.60.195:36339)
I20260810 07:24:45.699257 29521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:50795
I20260810 07:24:45.700269 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:24:45.708884 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.701713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.24.60.254:34043)
0810 07:24:45.701862 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.701864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.701878 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:24:45.701974 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.701975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.701987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.702052 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:45.702726 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.703233 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.704104 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.704314 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.704407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.706032 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:45.706034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:45.706034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:45.706043 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:45.707446 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.707898 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.707899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.707899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.707953 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:45.708273 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:45.708274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:45.708274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:45.708442 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:45.708567 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.708682 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.708772 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260810 07:24:45.711290 29521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:45.712980 29521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:45.720678 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.716047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36339 (local address 127.0.0.1:47518)
0810 07:24:45.716314 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.716322 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.716360 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.716922 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.716922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.716924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:45.717000 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.717002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.718016 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.718018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.719248 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.719251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.720353 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.720354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.720435 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.720452 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.720528 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.720583 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":36,"thread_start_us":95,"threads_started":1}
I20260810 07:24:45.720808 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.716386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.24.60.193:43311)
0810 07:24:45.716686 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.716691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.716702 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:45.716755 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.716757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.716774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.717090 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:45.717201 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.718295 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.719678 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.720390 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.720485 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.720542 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.720668 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.720717 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20260810 07:24:45.720806 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.715776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43311 (local address 127.0.0.1:43152)
0810 07:24:45.715965 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.716007 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.716077 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.716910 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.716911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.716913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:45.716980 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.716983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.718434 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.718435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.719496 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.719498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.720493 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.720494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.720560 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.720564 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.720647 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.720694 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":159,"threads_started":1}
I20260810 07:24:45.721236 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.716351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.24.60.195:36339)
0810 07:24:45.716688 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.716692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.716702 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:45.716765 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.716767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.716783 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.717084 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:45.717214 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.717908 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.719352 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.720260 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.720345 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.720416 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.721061 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.721107 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260810 07:24:45.721896 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.715776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:40175 (local address 127.0.0.1:36960)
0810 07:24:45.715990 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.716025 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.716112 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.716870 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.716873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.716886 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:45.717000 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.717004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.719361 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.719362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.720100 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.720102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.721608 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.721609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.721676 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.721680 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.721768 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.721816 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260810 07:24:45.721909 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.716008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.24.60.194:40175)
0810 07:24:45.716510 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.716516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.716550 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:24:45.716598 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.716600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.716619 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.716709 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:45.717093 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.718302 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.720203 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.721496 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.721602 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.721665 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.721785 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.721827 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":135,"threads_started":1}
I20260810 07:24:45.722043 29983 tablet_service.cc:1467] Processing CreateTablet for tablet c9c84beae5ff4e43b31fed6f72480291 (DEFAULT_TABLE table=test-table [id=8d1e67d30fe8446e8988aed4158735dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:45.722043 29667 tablet_service.cc:1467] Processing CreateTablet for tablet c9c84beae5ff4e43b31fed6f72480291 (DEFAULT_TABLE table=test-table [id=8d1e67d30fe8446e8988aed4158735dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:45.722453 29983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c84beae5ff4e43b31fed6f72480291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:45.722453 29667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c84beae5ff4e43b31fed6f72480291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:45.725813 29824 tablet_service.cc:1467] Processing CreateTablet for tablet c9c84beae5ff4e43b31fed6f72480291 (DEFAULT_TABLE table=test-table [id=8d1e67d30fe8446e8988aed4158735dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:45.726226 29824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c84beae5ff4e43b31fed6f72480291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:45.726337 30075 tablet_bootstrap.cc:492] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: Bootstrap starting.
I20260810 07:24:45.727027 30075 tablet_bootstrap.cc:654] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:45.727661 30075 log.cc:824] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:45.728108 30076 tablet_bootstrap.cc:492] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Bootstrap starting.
I20260810 07:24:45.728730 30076 tablet_bootstrap.cc:654] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:45.729298 30076 log.cc:824] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:45.729671 30075 tablet_bootstrap.cc:492] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: No bootstrap required, opened a new log
I20260810 07:24:45.729681 30078 tablet_bootstrap.cc:492] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Bootstrap starting.
I20260810 07:24:45.729795 30075 ts_tablet_manager.cc:1397] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:24:45.730281 30078 tablet_bootstrap.cc:654] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:45.730799 30078 log.cc:824] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:45.731698 30076 tablet_bootstrap.cc:492] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: No bootstrap required, opened a new log
I20260810 07:24:45.731621 30075 raft_consensus.cc:348] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.731775 30076 ts_tablet_manager.cc:1397] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:24:45.731809 30075 raft_consensus.cc:374] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:45.731868 30075 raft_consensus.cc:729] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fffd1e464746b59dc825b364b24562, State: Initialized, Role: FOLLOWER
I20260810 07:24:45.731995 30075 consensus_queue.cc:260] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.732366 30075 ts_tablet_manager.cc:1428] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:45.732419 29746 heartbeater.cc:503] Master 127.24.60.254:34043 was elected leader, sending a full tablet report...
I20260810 07:24:45.732824 30078 tablet_bootstrap.cc:492] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: No bootstrap required, opened a new log
I20260810 07:24:45.732903 30078 ts_tablet_manager.cc:1397] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:24:45.733985 30076 raft_consensus.cc:348] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.734159 30076 raft_consensus.cc:374] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:45.734194 30076 raft_consensus.cc:729] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f79d3b539b347f382d50c7eadb5fac4, State: Initialized, Role: FOLLOWER
I20260810 07:24:45.734320 30076 consensus_queue.cc:260] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.734645 30076 ts_tablet_manager.cc:1428] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:45.734665 30056 heartbeater.cc:503] Master 127.24.60.254:34043 was elected leader, sending a full tablet report...
I20260810 07:24:45.735018 30078 raft_consensus.cc:348] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.735170 30078 raft_consensus.cc:374] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:45.735216 30078 raft_consensus.cc:729] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75059f08ca041f7acc90ad43a68bb9e, State: Initialized, Role: FOLLOWER
I20260810 07:24:45.735375 30078 consensus_queue.cc:260] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.735695 29902 heartbeater.cc:503] Master 127.24.60.254:34043 was elected leader, sending a full tablet report...
I20260810 07:24:45.735695 30078 ts_tablet_manager.cc:1428] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260810 07:24:45.751922 29903 tablet.cc:2369] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:24:45.829787 29747 tablet.cc:2369] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:45.858495 30081 raft_consensus.cc:482] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:45.858637 30081 raft_consensus.cc:504] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.859010 30081 leader_election.cc:288] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f79d3b539b347f382d50c7eadb5fac4 (127.24.60.195:36339), f75059f08ca041f7acc90ad43a68bb9e (127.24.60.194:40175)
I20260810 07:24:45.862510 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.859342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53969 (local address 127.24.60.194:40175)
0810 07:24:45.859388 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.859390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.859515 (+ 125us) server_negotiation.cc:407] Connection header received
0810 07:24:45.859595 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.859596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.859607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.859688 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:45.859916 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.860415 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.861540 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.862103 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.862186 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.862196 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.862385 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.862430 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:24:45.862511 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.859301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:47265 (local address 127.24.60.195:36339)
0810 07:24:45.859366 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.859369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.859455 (+ 86us) server_negotiation.cc:407] Connection header received
0810 07:24:45.859550 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.859551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.859564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.859630 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:45.859794 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.860347 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.861505 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.862110 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.862186 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.862196 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.862384 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.862430 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:24:45.862533 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.859223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36339 (local address 127.24.60.193:47265)
0810 07:24:45.859379 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.859391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.859467 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.859636 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.859637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.859639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:45.859703 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.859706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.860452 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.860454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.861406 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.861409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.862198 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.862199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.862275 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.862279 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.862343 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.862409 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":46,"thread_start_us":69,"threads_started":1}
I20260810 07:24:45.862532 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.859260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:40175 (local address 127.24.60.193:53969)
0810 07:24:45.859418 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.859435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:45.859523 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:45.859694 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:45.859695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:45.859697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:45.859785 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.859787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.860494 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.860495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.861405 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:45.861408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.862196 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.862197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:45.862279 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.862283 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:45.862343 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 07:24:45.862439 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260810 07:24:45.862902 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c84beae5ff4e43b31fed6f72480291" candidate_uuid: "69fffd1e464746b59dc825b364b24562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f79d3b539b347f382d50c7eadb5fac4" is_pre_election: true
I20260810 07:24:45.863082 30005 raft_consensus.cc:2454] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fffd1e464746b59dc825b364b24562 in term 0.
I20260810 07:24:45.863363 29849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c84beae5ff4e43b31fed6f72480291" candidate_uuid: "69fffd1e464746b59dc825b364b24562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75059f08ca041f7acc90ad43a68bb9e" is_pre_election: true
I20260810 07:24:45.863512 29849 raft_consensus.cc:2454] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fffd1e464746b59dc825b364b24562 in term 0.
I20260810 07:24:45.863718 29614 leader_election.cc:302] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f79d3b539b347f382d50c7eadb5fac4, 69fffd1e464746b59dc825b364b24562; no voters:
I20260810 07:24:45.863924 30081 raft_consensus.cc:2781] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:45.864034 30081 raft_consensus.cc:482] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:45.864074 30081 raft_consensus.cc:3037] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:45.864805 30081 raft_consensus.cc:504] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.864981 30081 leader_election.cc:288] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [CANDIDATE]: Term 1 election: Requested vote from peers 3f79d3b539b347f382d50c7eadb5fac4 (127.24.60.195:36339), f75059f08ca041f7acc90ad43a68bb9e (127.24.60.194:40175)
I20260810 07:24:45.865209 29849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c84beae5ff4e43b31fed6f72480291" candidate_uuid: "69fffd1e464746b59dc825b364b24562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75059f08ca041f7acc90ad43a68bb9e"
I20260810 07:24:45.865298 29849 raft_consensus.cc:3037] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:45.866056 29849 raft_consensus.cc:2454] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fffd1e464746b59dc825b364b24562 in term 1.
I20260810 07:24:45.866185 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c84beae5ff4e43b31fed6f72480291" candidate_uuid: "69fffd1e464746b59dc825b364b24562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f79d3b539b347f382d50c7eadb5fac4"
I20260810 07:24:45.866279 29616 leader_election.cc:302] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fffd1e464746b59dc825b364b24562, f75059f08ca041f7acc90ad43a68bb9e; no voters:
I20260810 07:24:45.866353 30005 raft_consensus.cc:3037] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:45.866407 30081 raft_consensus.cc:2781] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:45.866611 30081 raft_consensus.cc:686] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [term 1 LEADER]: Becoming Leader. State: Replica: 69fffd1e464746b59dc825b364b24562, State: Running, Role: LEADER
I20260810 07:24:45.866727 30081 consensus_queue.cc:237] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } }
I20260810 07:24:45.867373 30005 raft_consensus.cc:2454] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fffd1e464746b59dc825b364b24562 in term 1.
I20260810 07:24:45.867708 29521 catalog_manager.cc:5495] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69fffd1e464746b59dc825b364b24562 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "69fffd1e464746b59dc825b364b24562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fffd1e464746b59dc825b364b24562" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:45.903214 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.900464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.24.60.193:43311)
0810 07:24:45.900534 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:45.900536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:45.900544 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:45.900585 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:45.900585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:45.900605 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:45.900696 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:45.901106 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.901646 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.902480 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:45.902681 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:45.902750 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:45.902914 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:45.903029 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:45.903103 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:45.903124 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 07:24:45.903145 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:24:45.905491 30005 raft_consensus.cc:1261] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 1 FOLLOWER]: Refusing update from remote peer 69fffd1e464746b59dc825b364b24562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:45.905491 29849 raft_consensus.cc:1261] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 1 FOLLOWER]: Refusing update from remote peer 69fffd1e464746b59dc825b364b24562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:45.906019 30081 consensus_queue.cc:1035] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75059f08ca041f7acc90ad43a68bb9e" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:45.906208 30086 consensus_queue.cc:1035] T c9c84beae5ff4e43b31fed6f72480291 P 69fffd1e464746b59dc825b364b24562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f79d3b539b347f382d50c7eadb5fac4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:45.909089 30092 mvcc.cc:205] Tried to move back new op lower bound from 7316876025465376768 to 7316876025310855168. Current Snapshot: MvccSnapshot[applied={T|T < 7316876025465376768}]
I20260810 07:24:45.909432 30090 mvcc.cc:205] Tried to move back new op lower bound from 7316876025465376768 to 7316876025310855168. Current Snapshot: MvccSnapshot[applied={T|T < 7316876025465376768}]
I20260810 07:24:45.912015 29521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:45.912140 29521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:45.913053 29521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bab123c717994394b42ccf9ec2afefbe: Sending DeleteTablet for 3 replicas of tablet c9c84beae5ff4e43b31fed6f72480291
I20260810 07:24:45.913462 29667 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c84beae5ff4e43b31fed6f72480291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:45 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:43152
I20260810 07:24:45.913524 29983 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c84beae5ff4e43b31fed6f72480291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:45 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260810 07:24:45.913517 29824 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c84beae5ff4e43b31fed6f72480291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:45 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:36960
I20260810 07:24:45.913743 30102 tablet_replica.cc:330] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: stopping tablet replica
I20260810 07:24:45.913861 30102 raft_consensus.cc:2229] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:45.913981 30102 raft_consensus.cc:2258] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:45.914160 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 29597
I20260810 07:24:45.915992 30102 ts_tablet_manager.cc:1905] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:45.918365 30102 ts_tablet_manager.cc:1918] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:45.918462 30102 log.cc:1196] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/c9c84beae5ff4e43b31fed6f72480291
I20260810 07:24:45.918746 30102 ts_tablet_manager.cc:1939] T c9c84beae5ff4e43b31fed6f72480291 P 3f79d3b539b347f382d50c7eadb5fac4: Deleting consensus metadata
I20260810 07:24:45.919567 30103 tablet_replica.cc:330] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: stopping tablet replica
I20260810 07:24:45.919605 29498 catalog_manager.cc:4841] TS 3f79d3b539b347f382d50c7eadb5fac4 (127.24.60.195:36339): tablet c9c84beae5ff4e43b31fed6f72480291 (table test-table [id=8d1e67d30fe8446e8988aed4158735dd]) successfully deleted
I20260810 07:24:45.919740 30103 raft_consensus.cc:2229] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:45.919893 30103 raft_consensus.cc:2258] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:45.920401 30103 ts_tablet_manager.cc:1905] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:45.922212 30103 ts_tablet_manager.cc:1918] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:45.922298 30103 log.cc:1196] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/c9c84beae5ff4e43b31fed6f72480291
I20260810 07:24:45.922576 30103 ts_tablet_manager.cc:1939] T c9c84beae5ff4e43b31fed6f72480291 P f75059f08ca041f7acc90ad43a68bb9e: Deleting consensus metadata
I20260810 07:24:45.923066 29500 catalog_manager.cc:4841] TS f75059f08ca041f7acc90ad43a68bb9e (127.24.60.194:40175): tablet c9c84beae5ff4e43b31fed6f72480291 (table test-table [id=8d1e67d30fe8446e8988aed4158735dd]) successfully deleted
W20260810 07:24:45.924654 29498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:43311 (error 108)
I20260810 07:24:45.925290 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:45.925117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:43311 (local address 127.0.0.1:43168)
0810 07:24:45.925201 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:45.925250 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 07:24:45.925303 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 29751
W20260810 07:24:45.925439 29498 catalog_manager.cc:4584] TS 69fffd1e464746b59dc825b364b24562 (127.24.60.193:43311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9c84beae5ff4e43b31fed6f72480291: Network error: Client connection negotiation failed: client connection to 127.24.60.193:43311: connect: Connection refused (error 111)
I20260810 07:24:45.934000 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 29907
I20260810 07:24:45.945309 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 29479
2026-08-10T07:24:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2253 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T07:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:45Z Disabled control of system clock
I20260810 07:24:45.977349 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:37245
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 07:24:46.061630 30110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:46.061950 30110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:46.061977 30110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:46.063828 30110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:46.063901 30110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:46.063916 30110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:46.063930 30110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:46.063943 30110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:46.065310 30110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:37245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:46.065521 30110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:46.065773 30110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:46.070994 30115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.070996 30116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.071218 30110 server_base.cc:1034] running on GCE node
W20260810 07:24:46.071094 30119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.071578 30110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:46.071897 30110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:46.073087 30110 hybrid_clock.cc:648] HybridClock initialized: now 1786346686073078 us; error 33 us; skew 500 ppm
I20260810 07:24:46.074498 30110 webserver.cc:460] Webserver started at http://127.24.60.254:33709/ using document root <none> and password file <none>
I20260810 07:24:46.074700 30110 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:46.074741 30110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:46.074838 30110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:46.075959 30110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "e95718a699014e65b325b190c7391249"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "e2c9b4e34724331ebd1d2e8926477328"
server_key_iv: "793627c15d58b1f7e6192220b9238e53"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.076264 30110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "e95718a699014e65b325b190c7391249"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "e2c9b4e34724331ebd1d2e8926477328"
server_key_iv: "793627c15d58b1f7e6192220b9238e53"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.077625 30110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:46.078414 30126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.078637 30110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:46.078714 30110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "e95718a699014e65b325b190c7391249"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "e2c9b4e34724331ebd1d2e8926477328"
server_key_iv: "793627c15d58b1f7e6192220b9238e53"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.078770 30110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:46.082257 30110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:46.082496 30110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:46.082594 30110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:46.086615 30110 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:37245
I20260810 07:24:46.086642 30187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:37245 every 8 connection(s)
I20260810 07:24:46.086999 30110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:46.087715 30188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:46.090569 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249: Bootstrap starting.
I20260810 07:24:46.091118 30188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:46.091357 30188 log.cc:824] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:46.092043 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249: No bootstrap required, opened a new log
I20260810 07:24:46.092414 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30110
I20260810 07:24:46.092530 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:46.092711 24819 external_mini_cluster.cc:1428] Setting key c8e39ec96d0e1934973704a30c6d5902
I20260810 07:24:46.094026 30188 raft_consensus.cc:348] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95718a699014e65b325b190c7391249" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37245 } }
I20260810 07:24:46.094204 30188 raft_consensus.cc:374] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:46.094246 30188 raft_consensus.cc:729] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95718a699014e65b325b190c7391249, State: Initialized, Role: FOLLOWER
I20260810 07:24:46.094421 30188 consensus_queue.cc:260] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95718a699014e65b325b190c7391249" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37245 } }
I20260810 07:24:46.094544 30188 raft_consensus.cc:388] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:46.094590 30188 raft_consensus.cc:482] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:46.094650 30188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:46.095399 30188 raft_consensus.cc:504] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95718a699014e65b325b190c7391249" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37245 } }
I20260810 07:24:46.095440 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.093299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.24.60.254:37245)
0810 07:24:46.093507 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.093515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.093539 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:24:46.094001 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.094008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.094047 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.094165 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.094957 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.094960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.094961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.094965 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.095009 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.095020 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.095077 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.095170 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.095239 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260810 07:24:46.095539 30188 leader_election.cc:302] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e95718a699014e65b325b190c7391249; no voters:
I20260810 07:24:46.096012 30194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:46.096045 30188 leader_election.cc:288] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:46.096158 30194 raft_consensus.cc:686] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [term 1 LEADER]: Becoming Leader. State: Replica: e95718a699014e65b325b190c7391249, State: Running, Role: LEADER
I20260810 07:24:46.096246 30194 consensus_queue.cc:237] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95718a699014e65b325b190c7391249" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37245 } }
I20260810 07:24:46.096288 30188 sys_catalog.cc:564] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:46.096577 30196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e95718a699014e65b325b190c7391249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95718a699014e65b325b190c7391249" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37245 } } }
I20260810 07:24:46.096577 30195 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e95718a699014e65b325b190c7391249. Latest consensus state: current_term: 1 leader_uuid: "e95718a699014e65b325b190c7391249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95718a699014e65b325b190c7391249" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37245 } } }
I20260810 07:24:46.096693 30196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:46.096709 30195 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:46.097004 30199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:46.097795 30199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:46.099444 30199 catalog_manager.cc:1347] Generated new cluster ID: 6c8b71a4112647398c37b85bbd125877
I20260810 07:24:46.099494 30199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:46.121906 30199 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:46.122123 30199 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:46.127126 30199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249: Generated new TSK 0
I20260810 07:24:46.127377 30199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:24:46.130118 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37245
--builtin_ntp_servers=127.24.60.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 07:24:46.218358 30215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:46.218639 30215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:46.218683 30215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:46.219975 30215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:46.220046 30215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:46.220091 30215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:46.221645 30215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:46.221943 30215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:46.222282 30215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:46.228636 30222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.228642 30221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.228683 30224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.229102 30215 server_base.cc:1034] running on GCE node
I20260810 07:24:46.229389 30215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:46.229702 30215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:46.230883 30215 hybrid_clock.cc:648] HybridClock initialized: now 1786346686230866 us; error 33 us; skew 500 ppm
I20260810 07:24:46.232458 30215 webserver.cc:460] Webserver started at http://127.24.60.193:42483/ using document root <none> and password file <none>
I20260810 07:24:46.232770 30215 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:46.232837 30215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:46.232981 30215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:46.234179 30215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "b1d814093fbd479194d6cc701ba16f32"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "63ef175cf7d7ea6abd9a2b0a4cd0e412"
server_key_iv: "1a0fa5afce273c86082d2c8e80d8e6d3"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.234489 30215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "b1d814093fbd479194d6cc701ba16f32"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "63ef175cf7d7ea6abd9a2b0a4cd0e412"
server_key_iv: "1a0fa5afce273c86082d2c8e80d8e6d3"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.235867 30215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:24:46.236730 30230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.236981 30215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:46.237098 30215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "b1d814093fbd479194d6cc701ba16f32"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "63ef175cf7d7ea6abd9a2b0a4cd0e412"
server_key_iv: "1a0fa5afce273c86082d2c8e80d8e6d3"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.237218 30215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:46.240707 30215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:46.241031 30215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:46.241219 30215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:46.241460 30215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:46.241809 30215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:46.241864 30215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.241904 30215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:46.241930 30215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.248231 30215 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36317
I20260810 07:24:46.248304 30357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36317 every 8 connection(s)
I20260810 07:24:46.248675 30215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:46.251966 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.249520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:48117 (local address 127.24.60.254:37245)
0810 07:24:46.249565 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.249568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.249807 (+ 239us) server_negotiation.cc:407] Connection header received
0810 07:24:46.250342 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.250344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.250354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.250421 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.251678 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.251680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.251681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.251686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.251704 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.251709 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.251771 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.251868 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.251889 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 07:24:46.252205 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.249507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:37245 (local address 127.24.60.193:48117)
0810 07:24:46.249677 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.249713 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.250226 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.250508 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.250512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.250520 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.251470 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.251471 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.251539 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.251570 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.251777 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.251782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.251785 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.251878 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.251947 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260810 07:24:46.252871 30358 heartbeater.cc:347] Connected to a master server at 127.24.60.254:37245
I20260810 07:24:46.252960 30358 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:46.253211 30358 heartbeater.cc:511] Master 127.24.60.254:37245 requested a full tablet report, sending...
I20260810 07:24:46.253881 30147 ts_manager.cc:194] Registered new tserver with Master: b1d814093fbd479194d6cc701ba16f32 (127.24.60.193:36317)
I20260810 07:24:46.256292 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30215
I20260810 07:24:46.256440 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:46.256675 24819 external_mini_cluster.cc:1428] Setting key 49c53d76ddfdc04097b0012066face38
I20260810 07:24:46.257817 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37245
--builtin_ntp_servers=127.24.60.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 07:24:46.347301 30361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:46.347553 30361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:46.347579 30361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:46.348781 30361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:46.348830 30361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:46.348858 30361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:46.350450 30361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:46.350716 30361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:46.351042 30361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:46.356796 30367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.356807 30370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.357091 30361 server_base.cc:1034] running on GCE node
W20260810 07:24:46.356817 30366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.357496 30361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:46.357790 30361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:46.358952 30361 hybrid_clock.cc:648] HybridClock initialized: now 1786346686358926 us; error 38 us; skew 500 ppm
I20260810 07:24:46.360406 30361 webserver.cc:460] Webserver started at http://127.24.60.194:37695/ using document root <none> and password file <none>
I20260810 07:24:46.360680 30361 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:46.360752 30361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:46.360966 30361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:46.362176 30361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "d0b710e152264e28a032fa331187bf7b"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "3270ded14c9a44e5543ebaa7bcc2674f"
server_key_iv: "64ce6c133366267ef2dcca541601ffcf"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.362565 30361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "d0b710e152264e28a032fa331187bf7b"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "3270ded14c9a44e5543ebaa7bcc2674f"
server_key_iv: "64ce6c133366267ef2dcca541601ffcf"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.364084 30361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:24:46.365123 30376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.365398 30361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:46.365509 30361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "d0b710e152264e28a032fa331187bf7b"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "3270ded14c9a44e5543ebaa7bcc2674f"
server_key_iv: "64ce6c133366267ef2dcca541601ffcf"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.365597 30361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:46.369364 30361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:46.369674 30361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:46.369827 30361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:46.370066 30361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:46.370483 30361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:46.370539 30361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.370582 30361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:46.370610 30361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.377003 30361 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:35135
I20260810 07:24:46.377238 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:35135 every 8 connection(s)
I20260810 07:24:46.377511 30361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:46.381299 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.378443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:59389 (local address 127.24.60.254:37245)
0810 07:24:46.378527 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.378530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.378869 (+ 339us) server_negotiation.cc:407] Connection header received
0810 07:24:46.379420 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.379421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.379428 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.379490 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.380808 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.380810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.380810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.380815 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.380834 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.380839 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.380995 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.381136 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.381216 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:24:46.381544 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.378435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:37245 (local address 127.24.60.194:59389)
0810 07:24:46.378683 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.378721 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.379301 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.379549 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.379552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.379561 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.380618 (+ 1057us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.380618 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.380654 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.380691 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.381009 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.381017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.381020 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.381112 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.381272 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260810 07:24:46.382387 30512 heartbeater.cc:347] Connected to a master server at 127.24.60.254:37245
I20260810 07:24:46.382496 30512 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:46.382822 30512 heartbeater.cc:511] Master 127.24.60.254:37245 requested a full tablet report, sending...
I20260810 07:24:46.383356 30147 ts_manager.cc:194] Registered new tserver with Master: d0b710e152264e28a032fa331187bf7b (127.24.60.194:35135)
I20260810 07:24:46.384006 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30361
I20260810 07:24:46.384092 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:46.384297 24819 external_mini_cluster.cc:1428] Setting key 185af4fb66b06ecf7e14908d96e84d65
I20260810 07:24:46.385502 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37245
--builtin_ntp_servers=127.24.60.212:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 07:24:46.473325 30515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:46.473562 30515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:46.473588 30515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:46.474990 30515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:46.475096 30515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:46.475149 30515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:46.476748 30515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:46.477084 30515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:46.477468 30515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:46.483165 30524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.483165 30523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.483244 30526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.483692 30515 server_base.cc:1034] running on GCE node
I20260810 07:24:46.483973 30515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:46.484201 30515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:46.485407 30515 hybrid_clock.cc:648] HybridClock initialized: now 1786346686485384 us; error 49 us; skew 500 ppm
I20260810 07:24:46.486876 30515 webserver.cc:460] Webserver started at http://127.24.60.195:41267/ using document root <none> and password file <none>
I20260810 07:24:46.487162 30515 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:46.487246 30515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:46.487428 30515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:46.488569 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "10596dbe8f1a40bce66929f1c5d99d6d"
server_key_iv: "611b788e1275c5899dd39cc3c367fc59"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.488960 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "10596dbe8f1a40bce66929f1c5d99d6d"
server_key_iv: "611b788e1275c5899dd39cc3c367fc59"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.490535 30515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:46.491391 30532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.491569 30515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:46.491681 30515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "10596dbe8f1a40bce66929f1c5d99d6d"
server_key_iv: "611b788e1275c5899dd39cc3c367fc59"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.491765 30515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:46.495446 30515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:46.495767 30515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:46.495926 30515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:46.496229 30515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:46.496672 30515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:46.496727 30515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.496771 30515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:46.496798 30515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.503515 30515 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:40383
I20260810 07:24:46.503558 30665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:40383 every 8 connection(s)
I20260810 07:24:46.503947 30515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:46.507462 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.504758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:51451 (local address 127.24.60.254:37245)
0810 07:24:46.504961 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.504963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.505188 (+ 225us) server_negotiation.cc:407] Connection header received
0810 07:24:46.505811 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.505813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.505821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.505873 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.507154 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.507156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.507156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.507162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.507179 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.507183 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.507243 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.507365 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.507384 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260810 07:24:46.507709 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.504751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:37245 (local address 127.24.60.195:51451)
0810 07:24:46.505034 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.505072 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.505669 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.505956 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.505960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.505969 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.506985 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.506986 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.507017 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.507049 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.507252 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.507258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.507261 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.507378 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.507447 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260810 07:24:46.508313 30666 heartbeater.cc:347] Connected to a master server at 127.24.60.254:37245
I20260810 07:24:46.508448 30666 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:46.508656 30666 heartbeater.cc:511] Master 127.24.60.254:37245 requested a full tablet report, sending...
I20260810 07:24:46.509207 30147 ts_manager.cc:194] Registered new tserver with Master: 4d58ec50b0ec4e9eaa6f18c1fffd804f (127.24.60.195:40383)
I20260810 07:24:46.511561 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30515
I20260810 07:24:46.511656 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:46.511862 24819 external_mini_cluster.cc:1428] Setting key 3a734794a5306a96cc4303dbeff3b747
I20260810 07:24:46.512894 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:24:46.515616 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.514317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.24.60.254:37245)
0810 07:24:46.514393 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.514396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.514594 (+ 198us) server_negotiation.cc:407] Connection header received
0810 07:24:46.514623 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.514624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.514630 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.514682 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.515373 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.515374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.515374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.515378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.515391 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.515395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.515442 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.515494 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.515506 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:24:46.517704 30147 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:46.519913 30147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:46.524634 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.522561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.24.60.195:40383)
0810 07:24:46.522744 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.522750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.522970 (+ 220us) server_negotiation.cc:407] Connection header received
0810 07:24:46.523094 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.523099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.523110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.523169 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.524174 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.524183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.524184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.524190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.524234 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.524267 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.524337 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.524519 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.524539 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260810 07:24:46.524626 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.522455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:40383 (local address 127.0.0.1:32946)
0810 07:24:46.522840 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.522894 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.523006 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.523553 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.523554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.523557 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.524053 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.524054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.524070 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.524091 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.524369 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.524377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.524380 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.524497 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.524511 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":91,"threads_started":1}
I20260810 07:24:46.524880 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.522966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.24.60.194:35135)
0810 07:24:46.523543 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.523547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.523557 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:46.523584 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.523586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.523595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.523677 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.524301 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.524308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.524308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.524312 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.524346 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.524381 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.524450 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.524789 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.524805 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":106,"threads_started":1}
I20260810 07:24:46.524947 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.522729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.24.60.193:36317)
0810 07:24:46.522956 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.522962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.523029 (+ 67us) server_negotiation.cc:407] Connection header received
0810 07:24:46.523094 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.523097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.523110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.523191 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.524487 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.524498 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.524499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.524504 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.524537 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.524566 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.524620 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.524842 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.524859 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":123,"threads_started":1}
I20260810 07:24:46.524631 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.522708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:35135 (local address 127.0.0.1:44502)
0810 07:24:46.523127 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.523138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.523171 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.523702 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.523703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.523707 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.524211 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.524212 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.524216 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.524223 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.524457 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.524458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.524459 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.524507 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.524522 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":77,"threads_started":1}
I20260810 07:24:46.524775 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.522466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36317 (local address 127.0.0.1:56958)
0810 07:24:46.522967 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.522977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.523034 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.523494 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.523500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.523506 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.524380 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.524380 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.524388 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.524399 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.524645 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.524646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.524646 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.524691 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.524705 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":121,"threads_started":1}
I20260810 07:24:46.525976 30284 tablet_service.cc:1467] Processing CreateTablet for tablet cc88661865594be49814c2908ade6f75 (DEFAULT_TABLE table=test-table [id=2c1593383d3341f8b2ea308e32f4f50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:46.526006 30428 tablet_service.cc:1467] Processing CreateTablet for tablet cc88661865594be49814c2908ade6f75 (DEFAULT_TABLE table=test-table [id=2c1593383d3341f8b2ea308e32f4f50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:46.526003 30588 tablet_service.cc:1467] Processing CreateTablet for tablet cc88661865594be49814c2908ade6f75 (DEFAULT_TABLE table=test-table [id=2c1593383d3341f8b2ea308e32f4f50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:46.526356 30284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc88661865594be49814c2908ade6f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:46.526360 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc88661865594be49814c2908ade6f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:46.526369 30428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc88661865594be49814c2908ade6f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:46.529238 30683 tablet_bootstrap.cc:492] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32: Bootstrap starting.
I20260810 07:24:46.529858 30684 tablet_bootstrap.cc:492] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Bootstrap starting.
I20260810 07:24:46.530234 30683 tablet_bootstrap.cc:654] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:46.530412 30685 tablet_bootstrap.cc:492] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Bootstrap starting.
I20260810 07:24:46.530599 30684 tablet_bootstrap.cc:654] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:46.530813 30683 log.cc:824] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:46.530941 30684 log.cc:824] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:46.531116 30685 tablet_bootstrap.cc:654] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:46.531483 30685 log.cc:824] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:46.531694 30683 tablet_bootstrap.cc:492] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32: No bootstrap required, opened a new log
I20260810 07:24:46.531776 30683 ts_tablet_manager.cc:1397] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:24:46.531847 30684 tablet_bootstrap.cc:492] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: No bootstrap required, opened a new log
I20260810 07:24:46.531920 30684 ts_tablet_manager.cc:1397] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:46.532251 30685 tablet_bootstrap.cc:492] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: No bootstrap required, opened a new log
I20260810 07:24:46.532354 30685 ts_tablet_manager.cc:1397] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:46.534585 30685 raft_consensus.cc:348] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.534775 30685 raft_consensus.cc:374] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:46.534822 30685 raft_consensus.cc:729] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d58ec50b0ec4e9eaa6f18c1fffd804f, State: Initialized, Role: FOLLOWER
I20260810 07:24:46.534794 30683 raft_consensus.cc:348] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.534989 30683 raft_consensus.cc:374] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:46.535028 30683 raft_consensus.cc:729] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d814093fbd479194d6cc701ba16f32, State: Initialized, Role: FOLLOWER
I20260810 07:24:46.534967 30684 raft_consensus.cc:348] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.535125 30684 raft_consensus.cc:374] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:46.535159 30684 raft_consensus.cc:729] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b710e152264e28a032fa331187bf7b, State: Initialized, Role: FOLLOWER
I20260810 07:24:46.535178 30683 consensus_queue.cc:260] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.535202 30685 consensus_queue.cc:260] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.535290 30684 consensus_queue.cc:260] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.535498 30683 ts_tablet_manager.cc:1428] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 07:24:46.535567 30684 ts_tablet_manager.cc:1428] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 07:24:46.535534 30685 ts_tablet_manager.cc:1428] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:46.535640 30358 heartbeater.cc:503] Master 127.24.60.254:37245 was elected leader, sending a full tablet report...
I20260810 07:24:46.535707 30666 heartbeater.cc:503] Master 127.24.60.254:37245 was elected leader, sending a full tablet report...
I20260810 07:24:46.535709 30512 heartbeater.cc:503] Master 127.24.60.254:37245 was elected leader, sending a full tablet report...
I20260810 07:24:46.556684 30689 raft_consensus.cc:482] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:46.556936 30689 raft_consensus.cc:504] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.557358 30689 leader_election.cc:288] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d58ec50b0ec4e9eaa6f18c1fffd804f (127.24.60.195:40383), d0b710e152264e28a032fa331187bf7b (127.24.60.194:35135)
I20260810 07:24:46.559149 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.557595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:35135 (local address 127.24.60.193:58341)
0810 07:24:46.557672 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.557686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.557730 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.558086 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.558087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.558090 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.558619 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.558620 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.558630 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.558647 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.558984 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.558985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.558986 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.559050 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.559067 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 07:24:46.559221 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.557809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58341 (local address 127.24.60.194:35135)
0810 07:24:46.557947 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.557950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.557959 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:46.557984 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.557985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.557990 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.558073 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.558840 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.558841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.558842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.558849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.558870 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.558876 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.558950 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.559117 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.559134 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 07:24:46.559297 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.557691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:51343 (local address 127.24.60.195:40383)
0810 07:24:46.557751 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.557753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.558021 (+ 268us) server_negotiation.cc:407] Connection header received
0810 07:24:46.558099 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.558101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.558106 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.558160 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.559000 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.559002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.559002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.559007 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.559028 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.559033 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.559102 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.559197 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.559214 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 07:24:46.559501 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.557595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:40383 (local address 127.24.60.193:51343)
0810 07:24:46.557935 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.557946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.558030 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.558181 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.558182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.558184 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:46.558798 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:24:46.558798 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:46.558808 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:24:46.558824 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:46.559120 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:46.559122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:46.559123 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 07:24:46.559188 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:24:46.559430 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":128,"thread_start_us":140,"threads_started":1}
I20260810 07:24:46.559581 30452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc88661865594be49814c2908ade6f75" candidate_uuid: "b1d814093fbd479194d6cc701ba16f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b710e152264e28a032fa331187bf7b" is_pre_election: true
I20260810 07:24:46.559762 30452 raft_consensus.cc:2454] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1d814093fbd479194d6cc701ba16f32 in term 0.
I20260810 07:24:46.559851 30613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc88661865594be49814c2908ade6f75" candidate_uuid: "b1d814093fbd479194d6cc701ba16f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" is_pre_election: true
I20260810 07:24:46.560007 30613 raft_consensus.cc:2454] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1d814093fbd479194d6cc701ba16f32 in term 0.
I20260810 07:24:46.560196 30235 leader_election.cc:302] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1d814093fbd479194d6cc701ba16f32, d0b710e152264e28a032fa331187bf7b; no voters:
I20260810 07:24:46.560361 30689 raft_consensus.cc:2781] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:46.560468 30689 raft_consensus.cc:482] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:46.560516 30689 raft_consensus.cc:3037] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:46.561448 30689 raft_consensus.cc:504] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.561631 30689 leader_election.cc:288] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [CANDIDATE]: Term 1 election: Requested vote from peers 4d58ec50b0ec4e9eaa6f18c1fffd804f (127.24.60.195:40383), d0b710e152264e28a032fa331187bf7b (127.24.60.194:35135)
I20260810 07:24:46.561808 30613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc88661865594be49814c2908ade6f75" candidate_uuid: "b1d814093fbd479194d6cc701ba16f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f"
I20260810 07:24:46.561844 30452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc88661865594be49814c2908ade6f75" candidate_uuid: "b1d814093fbd479194d6cc701ba16f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b710e152264e28a032fa331187bf7b"
I20260810 07:24:46.561910 30613 raft_consensus.cc:3037] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:46.561971 30452 raft_consensus.cc:3037] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:46.562769 30613 raft_consensus.cc:2454] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1d814093fbd479194d6cc701ba16f32 in term 1.
I20260810 07:24:46.562777 30452 raft_consensus.cc:2454] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1d814093fbd479194d6cc701ba16f32 in term 1.
I20260810 07:24:46.563010 30235 leader_election.cc:302] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1d814093fbd479194d6cc701ba16f32, d0b710e152264e28a032fa331187bf7b; no voters:
I20260810 07:24:46.563154 30689 raft_consensus.cc:2781] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:46.563431 30689 raft_consensus.cc:686] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [term 1 LEADER]: Becoming Leader. State: Replica: b1d814093fbd479194d6cc701ba16f32, State: Running, Role: LEADER
I20260810 07:24:46.563568 30689 consensus_queue.cc:237] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } }
I20260810 07:24:46.564513 30144 catalog_manager.cc:5495] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1d814093fbd479194d6cc701ba16f32 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "b1d814093fbd479194d6cc701ba16f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1d814093fbd479194d6cc701ba16f32" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:46.577628 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.576224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.24.60.193:36317)
0810 07:24:46.576309 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.576311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.576330 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:24:46.576373 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.576374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.576390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.576461 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.577219 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.577222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.577222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.577228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.577254 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.577259 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.577328 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.577489 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.577511 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:24:46.579908 30613 raft_consensus.cc:1261] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 1 FOLLOWER]: Refusing update from remote peer b1d814093fbd479194d6cc701ba16f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:46.580013 30452 raft_consensus.cc:1261] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 1 FOLLOWER]: Refusing update from remote peer b1d814093fbd479194d6cc701ba16f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:46.580341 30689 consensus_queue.cc:1035] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d58ec50b0ec4e9eaa6f18c1fffd804f" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:46.580538 30694 consensus_queue.cc:1035] T cc88661865594be49814c2908ade6f75 P b1d814093fbd479194d6cc701ba16f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b710e152264e28a032fa331187bf7b" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:46.583554 30697 mvcc.cc:205] Tried to move back new op lower bound from 7316876028228136960 to 7316876028165140480. Current Snapshot: MvccSnapshot[applied={T|T < 7316876028228136960}]
I20260810 07:24:46.584290 30699 mvcc.cc:205] Tried to move back new op lower bound from 7316876028228136960 to 7316876028165140480. Current Snapshot: MvccSnapshot[applied={T|T < 7316876028228136960}]
I20260810 07:24:46.587227 30144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:46.587345 30144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:46.588316 30144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e95718a699014e65b325b190c7391249: Sending DeleteTablet for 3 replicas of tablet cc88661865594be49814c2908ade6f75
I20260810 07:24:46.588734 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet cc88661865594be49814c2908ade6f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:46 UTC) from {username='slave'} at 127.0.0.1:32946
I20260810 07:24:46.588727 30284 tablet_service.cc:1514] Processing DeleteTablet for tablet cc88661865594be49814c2908ade6f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:46 UTC) from {username='slave'} at 127.0.0.1:56958
I20260810 07:24:46.589118 30428 tablet_service.cc:1514] Processing DeleteTablet for tablet cc88661865594be49814c2908ade6f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:46 UTC) from {username='slave'} at 127.0.0.1:44502
I20260810 07:24:46.589224 30709 tablet_replica.cc:330] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: stopping tablet replica
I20260810 07:24:46.589499 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30215
I20260810 07:24:46.589375 30709 raft_consensus.cc:2229] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:46.591626 30709 raft_consensus.cc:2258] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:46.592347 30712 tablet_replica.cc:330] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: stopping tablet replica
I20260810 07:24:46.592512 30712 raft_consensus.cc:2229] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:46.592649 30712 raft_consensus.cc:2258] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:46.593257 30712 ts_tablet_manager.cc:1905] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:46.593335 30709 ts_tablet_manager.cc:1905] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:46.595664 30709 ts_tablet_manager.cc:1918] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:46.595801 30709 log.cc:1196] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/cc88661865594be49814c2908ade6f75
I20260810 07:24:46.596096 30709 ts_tablet_manager.cc:1939] T cc88661865594be49814c2908ade6f75 P 4d58ec50b0ec4e9eaa6f18c1fffd804f: Deleting consensus metadata
I20260810 07:24:46.596689 30712 ts_tablet_manager.cc:1918] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:46.596815 30712 log.cc:1196] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/cc88661865594be49814c2908ade6f75
I20260810 07:24:46.597211 30712 ts_tablet_manager.cc:1939] T cc88661865594be49814c2908ade6f75 P d0b710e152264e28a032fa331187bf7b: Deleting consensus metadata
I20260810 07:24:46.597348 30128 catalog_manager.cc:4841] TS 4d58ec50b0ec4e9eaa6f18c1fffd804f (127.24.60.195:40383): tablet cc88661865594be49814c2908ade6f75 (table test-table [id=2c1593383d3341f8b2ea308e32f4f50d]) successfully deleted
I20260810 07:24:46.597757 30131 catalog_manager.cc:4841] TS d0b710e152264e28a032fa331187bf7b (127.24.60.194:35135): tablet cc88661865594be49814c2908ade6f75 (table test-table [id=2c1593383d3341f8b2ea308e32f4f50d]) successfully deleted
W20260810 07:24:46.601818 30131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:36317 (error 108)
I20260810 07:24:46.602528 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.602337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36317 (local address 127.0.0.1:56984)
0810 07:24:46.602441 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.602494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 07:24:46.602612 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30361
W20260810 07:24:46.602695 30131 catalog_manager.cc:4584] TS b1d814093fbd479194d6cc701ba16f32 (127.24.60.193:36317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc88661865594be49814c2908ade6f75: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36317: connect: Connection refused (error 111)
I20260810 07:24:46.609496 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30515
I20260810 07:24:46.618777 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30110
2026-08-10T07:24:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (673 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 07:24:46.642261 24819 mini_oidc.cc:152] Starting JWKS server
I20260810 07:24:46.643338 24819 webserver.cc:460] Webserver started at https://127.0.0.1:37531/ using document root <none> and password file <none>
I20260810 07:24:46.643489 24819 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 07:24:46.643708 24819 webserver.cc:460] Webserver started at http://127.0.0.1:42141/ using document root <none> and password file <none>
2026-08-10T07:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:46Z Disabled control of system clock
I20260810 07:24:46.652657 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34067
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:34067
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/testchainca.cert with env {}
W20260810 07:24:46.739789 30719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:46.740065 30719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:46.740095 30719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:46.741581 30719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:46.741644 30719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:46.741657 30719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:46.741671 30719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:46.741683 30719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:24:46.741698 30719 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:24:46.741708 30719 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37531/jwks.json
I20260810 07:24:46.743613 30719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:34067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:46.743947 30719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:46.744231 30719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:46.749830 30727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.749918 30725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.749931 30724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.750278 30719 server_base.cc:1034] running on GCE node
I20260810 07:24:46.750558 30719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:46.750881 30719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:46.752060 30719 hybrid_clock.cc:648] HybridClock initialized: now 1786346686752048 us; error 37 us; skew 500 ppm
I20260810 07:24:46.753692 30719 webserver.cc:460] Webserver started at http://127.24.60.254:43573/ using document root <none> and password file <none>
I20260810 07:24:46.754012 30719 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:46.754104 30719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:46.754294 30719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:46.755486 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "af0ccb5310044937b112d6a373357e0c"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "fe1d817b1105b9e965fd36c614baed3d"
server_key_iv: "cf1478bc135467cbc06f804b6762f267"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.755895 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "af0ccb5310044937b112d6a373357e0c"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "fe1d817b1105b9e965fd36c614baed3d"
server_key_iv: "cf1478bc135467cbc06f804b6762f267"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.757395 30719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260810 07:24:46.758323 30733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.758555 30719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:46.758672 30719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "af0ccb5310044937b112d6a373357e0c"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "fe1d817b1105b9e965fd36c614baed3d"
server_key_iv: "cf1478bc135467cbc06f804b6762f267"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.758780 30719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:46.770241 30719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:46.770604 30719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:46.770772 30719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:46.775244 30719 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:34067
I20260810 07:24:46.775292 30790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:34067 every 8 connection(s)
I20260810 07:24:46.775679 30719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:46.776724 30792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:46.777910 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30719
I20260810 07:24:46.778056 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:46.778287 24819 external_mini_cluster.cc:1428] Setting key d437ab513b2f93c34fd71cec3e90c717
I20260810 07:24:46.779702 30792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c: Bootstrap starting.
I20260810 07:24:46.780331 30792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:46.780654 30792 log.cc:824] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:46.781524 30792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c: No bootstrap required, opened a new log
I20260810 07:24:46.783420 30792 raft_consensus.cc:348] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0ccb5310044937b112d6a373357e0c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34067 } }
I20260810 07:24:46.783602 30792 raft_consensus.cc:374] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:46.783650 30792 raft_consensus.cc:729] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0ccb5310044937b112d6a373357e0c, State: Initialized, Role: FOLLOWER
I20260810 07:24:46.783823 30792 consensus_queue.cc:260] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0ccb5310044937b112d6a373357e0c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34067 } }
I20260810 07:24:46.783926 30792 raft_consensus.cc:388] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:46.783969 30792 raft_consensus.cc:482] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:46.784022 30792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:46.784643 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.778986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.24.60.254:34067)
0810 07:24:46.779223 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.779231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.779284 (+ 53us) server_negotiation.cc:407] Connection header received
0810 07:24:46.779738 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.779745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.779835 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.780009 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:46.780280 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:46.781461 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:46.782347 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:46.782820 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:46.782970 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:46.783813 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:46.783817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:46.783817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:46.784074 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:46.784144 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:46.784160 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:46.784231 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:46.784330 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 07:24:46.784418 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260810 07:24:46.784801 30792 raft_consensus.cc:504] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0ccb5310044937b112d6a373357e0c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34067 } }
I20260810 07:24:46.784924 30792 leader_election.cc:302] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0ccb5310044937b112d6a373357e0c; no voters:
I20260810 07:24:46.785105 30792 leader_election.cc:288] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:46.785197 30799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:46.785352 30799 raft_consensus.cc:686] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [term 1 LEADER]: Becoming Leader. State: Replica: af0ccb5310044937b112d6a373357e0c, State: Running, Role: LEADER
I20260810 07:24:46.785418 30792 sys_catalog.cc:564] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:46.785451 30799 consensus_queue.cc:237] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0ccb5310044937b112d6a373357e0c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34067 } }
I20260810 07:24:46.785799 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:24:46.785827 30799 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader af0ccb5310044937b112d6a373357e0c. Latest consensus state: current_term: 1 leader_uuid: "af0ccb5310044937b112d6a373357e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0ccb5310044937b112d6a373357e0c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34067 } } }
I20260810 07:24:46.785899 30799 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:46.786139 30805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:46.786952 30801 sys_catalog.cc:455] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af0ccb5310044937b112d6a373357e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0ccb5310044937b112d6a373357e0c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34067 } } }
I20260810 07:24:46.787058 30801 sys_catalog.cc:458] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:46.787020 30805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:46.789072 30805 catalog_manager.cc:1347] Generated new cluster ID: 645de96c43354a968f61495c894fc18a
I20260810 07:24:46.789139 30805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:46.799405 30805 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:46.800083 30805 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:46.807192 30805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af0ccb5310044937b112d6a373357e0c: Generated new TSK 0
I20260810 07:24:46.807420 30805 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 07:24:46.834888 30831 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 07:24:46.835036 30831 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 07:24:46.835258 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:46.819294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.24.60.254:34067)
0810 07:24:46.819348 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:46.819351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:46.819538 (+ 187us) server_negotiation.cc:407] Connection header received
0810 07:24:46.819674 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:46.819675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:46.819687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:46.819768 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:24:46.820009 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:46.820602 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:46.821796 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:46.822006 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:46.822065 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:46.822342 (+ 277us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:24:46.834986 (+ 12644us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 07:24:46.835173 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 07:24:46.835340 30796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 07:24:46.835677 30830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:46.819183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34067 (local address 127.0.0.1:43642)
0810 07:24:46.819412 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:46.819426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:46.819519 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:46.819805 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:46.819807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:46.819809 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:24:46.819889 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:46.819892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:46.820711 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:46.820712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:46.821700 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:46.821705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:46.822113 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:46.822115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:46.822185 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:46.822212 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:24:46.835249 (+ 13037us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 07:24:46.835382 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.60.254:34067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.60.254:34067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 07:24:46.836359 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30719
2026-08-10T07:24:46Z chronyd exiting
I20260810 07:24:46.875597 24819 test_util.cc:175] -----------------------------------------------
I20260810 07:24:46.875747 24819 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346663269703-24819-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 07:24:46.876711 24819 mini_oidc.cc:152] Starting JWKS server
I20260810 07:24:46.877647 24819 webserver.cc:460] Webserver started at https://127.0.0.1:34591/ using document root <none> and password file <none>
I20260810 07:24:46.877697 24819 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 07:24:46.877894 24819 webserver.cc:460] Webserver started at http://127.0.0.1:33577/ using document root <none> and password file <none>
2026-08-10T07:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:46Z Disabled control of system clock
I20260810 07:24:46.888212 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44853
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:44853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34591/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/testchainca.cert with env {}
W20260810 07:24:46.974210 30839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:46.974447 30839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:46.974472 30839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:46.975657 30839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:46.975697 30839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:46.975709 30839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:46.975723 30839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:46.975734 30839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:24:46.975746 30839 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:24:46.975757 30839 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34591/jwks.json
I20260810 07:24:46.977311 30839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:44853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34591/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:46.977598 30839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:46.977856 30839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:46.983569 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.983573 30846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:46.983570 30845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:46.984030 30839 server_base.cc:1034] running on GCE node
I20260810 07:24:46.984278 30839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:46.984565 30839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:46.985772 30839 hybrid_clock.cc:648] HybridClock initialized: now 1786346686985761 us; error 41 us; skew 500 ppm
I20260810 07:24:46.987449 30839 webserver.cc:460] Webserver started at http://127.24.60.254:43591/ using document root <none> and password file <none>
I20260810 07:24:46.987758 30839 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:46.987901 30839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:46.988075 30839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:46.989395 30839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "60923f81d81d4557abd91ebe8ef8cc44"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "03e29094fb680488680dc046c603b5d3"
server_key_iv: "b8f4743c52e34e8b139940b784dc13e1"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.989802 30839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "60923f81d81d4557abd91ebe8ef8cc44"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "03e29094fb680488680dc046c603b5d3"
server_key_iv: "b8f4743c52e34e8b139940b784dc13e1"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.991325 30839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:46.992209 30856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:46.992412 30839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:46.992532 30839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "60923f81d81d4557abd91ebe8ef8cc44"
format_stamp: "Formatted at 2026-08-10 07:24:46 on dist-test-slave-vxj2"
server_key: "03e29094fb680488680dc046c603b5d3"
server_key_iv: "b8f4743c52e34e8b139940b784dc13e1"
server_key_version: "encryptionkey@0"
I20260810 07:24:46.992622 30839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:47.006349 30839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:47.006807 30839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:47.007012 30839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:47.012097 30839 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:44853
I20260810 07:24:47.012141 30918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:44853 every 8 connection(s)
I20260810 07:24:47.012570 30839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:47.013451 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:47.013612 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30839
I20260810 07:24:47.013747 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:47.013981 24819 external_mini_cluster.cc:1428] Setting key 29c8babed1422ea24227ea6cec299ff9
I20260810 07:24:47.017824 30919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44: Bootstrap starting.
I20260810 07:24:47.018556 30919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:47.018898 30919 log.cc:824] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:47.020102 30919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44: No bootstrap required, opened a new log
I20260810 07:24:47.020542 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:47.014647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.24.60.254:44853)
0810 07:24:47.014887 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:47.014895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:47.014924 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:24:47.015383 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:47.015392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:47.015477 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:47.015614 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:47.015955 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.016890 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.018106 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.018590 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.018756 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.019673 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:47.019679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:47.019679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:47.019907 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:47.019955 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:47.019973 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:47.020056 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:47.020179 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 07:24:47.020295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260810 07:24:47.021811 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:24:47.022490 30919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60923f81d81d4557abd91ebe8ef8cc44" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44853 } }
I20260810 07:24:47.022693 30919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:47.022734 30919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60923f81d81d4557abd91ebe8ef8cc44, State: Initialized, Role: FOLLOWER
I20260810 07:24:47.022868 30919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60923f81d81d4557abd91ebe8ef8cc44" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44853 } }
I20260810 07:24:47.022951 30919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:47.022979 30919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:47.023048 30919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:47.023711 30919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60923f81d81d4557abd91ebe8ef8cc44" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44853 } }
I20260810 07:24:47.023805 30919 leader_election.cc:302] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60923f81d81d4557abd91ebe8ef8cc44; no voters:
I20260810 07:24:47.023959 30919 leader_election.cc:288] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:47.024019 30924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:47.024210 30919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:47.024253 30924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [term 1 LEADER]: Becoming Leader. State: Replica: 60923f81d81d4557abd91ebe8ef8cc44, State: Running, Role: LEADER
I20260810 07:24:47.024408 30924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60923f81d81d4557abd91ebe8ef8cc44" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44853 } }
I20260810 07:24:47.024873 30926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60923f81d81d4557abd91ebe8ef8cc44. Latest consensus state: current_term: 1 leader_uuid: "60923f81d81d4557abd91ebe8ef8cc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60923f81d81d4557abd91ebe8ef8cc44" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44853 } } }
I20260810 07:24:47.025002 30926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:47.024861 30925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60923f81d81d4557abd91ebe8ef8cc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60923f81d81d4557abd91ebe8ef8cc44" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44853 } } }
I20260810 07:24:47.025350 30925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:47.025281 30934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:47.026247 30934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:47.027976 30934 catalog_manager.cc:1347] Generated new cluster ID: a6c7587a6a9d431bb77820c6f7ad71b7
I20260810 07:24:47.028053 30934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:47.043812 30934 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:47.044445 30934 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:47.048574 30934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60923f81d81d4557abd91ebe8ef8cc44: Generated new TSK 0
I20260810 07:24:47.048792 30934 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 07:24:47.076372 30951 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 07:24:47.076504 30951 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 07:24:47.076870 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:47.061360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.24.60.254:44853)
0810 07:24:47.061432 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:47.061435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:47.061601 (+ 166us) server_negotiation.cc:407] Connection header received
0810 07:24:47.061693 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:47.061694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:47.061709 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:47.061788 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:24:47.061989 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.062798 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.063641 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.063972 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.064043 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.064241 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:24:47.076616 (+ 12375us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 07:24:47.076793 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 07:24:47.076951 30922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 07:24:47.076972 30950 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:47.061283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44853 (local address 127.0.0.1:35038)
0810 07:24:47.061518 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:47.061534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:47.061598 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:47.061796 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:47.061797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:47.061799 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:24:47.061864 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:47.061866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.062934 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.062935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:47.063549 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:47.063551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.064059 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.064060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:47.064115 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.064121 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:24:47.076751 (+ 12630us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 07:24:47.076855 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.60.254:44853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34591/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":161,"threads_started":1}
I20260810 07:24:47.077641 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30839
2026-08-10T07:24:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (243 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 07:24:47.119850 24819 mini_oidc.cc:152] Starting JWKS server
I20260810 07:24:47.120546 24819 webserver.cc:460] Webserver started at https://127.0.0.1:42491/ using document root <none> and password file <none>
I20260810 07:24:47.120633 24819 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 07:24:47.120821 24819 webserver.cc:460] Webserver started at http://127.0.0.1:43231/ using document root <none> and password file <none>
2026-08-10T07:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:47Z Disabled control of system clock
I20260810 07:24:47.129607 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33989
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 07:24:47.217775 30957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:47.218075 30957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:47.218114 30957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:47.219357 30957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:47.219447 30957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:47.219473 30957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:47.219497 30957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:47.219525 30957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:24:47.219550 30957 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:24:47.219571 30957 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42491/jwks.json
I20260810 07:24:47.221138 30957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:47.221480 30957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:47.221805 30957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:47.227298 30966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:47.227371 30963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:47.227429 30964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:47.227680 30957 server_base.cc:1034] running on GCE node
I20260810 07:24:47.227970 30957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:47.228294 30957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:47.229475 30957 hybrid_clock.cc:648] HybridClock initialized: now 1786346687229451 us; error 36 us; skew 500 ppm
I20260810 07:24:47.230938 30957 webserver.cc:460] Webserver started at http://127.24.60.254:33309/ using document root <none> and password file <none>
I20260810 07:24:47.231201 30957 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:47.231272 30957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:47.231465 30957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:47.232648 30957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "f00c011c93744c5b9744e275e23f57e2"
format_stamp: "Formatted at 2026-08-10 07:24:47 on dist-test-slave-vxj2"
server_key: "cd7bc873c3b5561c1870d699bd4184db"
server_key_iv: "d562730f6b2c300be4e1cc8e2b77a126"
server_key_version: "encryptionkey@0"
I20260810 07:24:47.233027 30957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "f00c011c93744c5b9744e275e23f57e2"
format_stamp: "Formatted at 2026-08-10 07:24:47 on dist-test-slave-vxj2"
server_key: "cd7bc873c3b5561c1870d699bd4184db"
server_key_iv: "d562730f6b2c300be4e1cc8e2b77a126"
server_key_version: "encryptionkey@0"
I20260810 07:24:47.234541 30957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:47.235437 30973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:47.235675 30957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:47.235798 30957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "f00c011c93744c5b9744e275e23f57e2"
format_stamp: "Formatted at 2026-08-10 07:24:47 on dist-test-slave-vxj2"
server_key: "cd7bc873c3b5561c1870d699bd4184db"
server_key_iv: "d562730f6b2c300be4e1cc8e2b77a126"
server_key_version: "encryptionkey@0"
I20260810 07:24:47.235879 30957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:47.248831 30957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:47.249233 30957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:47.249436 30957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:47.254076 30957 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:33989
I20260810 07:24:47.254139 31035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:33989 every 8 connection(s)
I20260810 07:24:47.254537 30957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:47.255087 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 30957
I20260810 07:24:47.255213 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:47.255450 24819 external_mini_cluster.cc:1428] Setting key e751e259e99f7c36325afcb3976baef1
I20260810 07:24:47.255470 31036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:47.259334 31036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2: Bootstrap starting.
I20260810 07:24:47.259996 31036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:47.260321 31036 log.cc:824] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:47.261121 31036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2: No bootstrap required, opened a new log
I20260810 07:24:47.262326 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:47.256352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.24.60.254:33989)
0810 07:24:47.256562 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:47.256572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:47.256606 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:24:47.257153 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:47.257163 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:47.257296 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:47.257459 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:47.257693 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.259164 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.260080 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.260694 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.260856 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.261660 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:47.261665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:47.261665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:47.261850 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:47.261885 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:47.261896 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:47.261977 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:47.262059 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 07:24:47.262144 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260810 07:24:47.263513 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:24:47.263437 31036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00c011c93744c5b9744e275e23f57e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33989 } }
I20260810 07:24:47.263633 31036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:47.263698 31036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00c011c93744c5b9744e275e23f57e2, State: Initialized, Role: FOLLOWER
I20260810 07:24:47.263849 31036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00c011c93744c5b9744e275e23f57e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33989 } }
I20260810 07:24:47.263948 31036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:47.263993 31036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:47.264043 31036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:47.264796 31036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00c011c93744c5b9744e275e23f57e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33989 } }
I20260810 07:24:47.264963 31036 leader_election.cc:302] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f00c011c93744c5b9744e275e23f57e2; no voters:
I20260810 07:24:47.265215 31036 leader_election.cc:288] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:47.265262 31042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:47.265478 31036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:47.265491 31042 raft_consensus.cc:686] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [term 1 LEADER]: Becoming Leader. State: Replica: f00c011c93744c5b9744e275e23f57e2, State: Running, Role: LEADER
I20260810 07:24:47.265676 31042 consensus_queue.cc:237] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00c011c93744c5b9744e275e23f57e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33989 } }
I20260810 07:24:47.266113 31043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f00c011c93744c5b9744e275e23f57e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00c011c93744c5b9744e275e23f57e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33989 } } }
I20260810 07:24:47.266145 31044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f00c011c93744c5b9744e275e23f57e2. Latest consensus state: current_term: 1 leader_uuid: "f00c011c93744c5b9744e275e23f57e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00c011c93744c5b9744e275e23f57e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33989 } } }
I20260810 07:24:47.266300 31044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:47.266557 31051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:47.266786 31043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:47.267102 31051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:47.268968 31051 catalog_manager.cc:1347] Generated new cluster ID: 4ef8aad83d4d418490d35d72b9f94024
I20260810 07:24:47.269027 31051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:47.278250 31051 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:47.278836 31051 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:47.281929 31051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f00c011c93744c5b9744e275e23f57e2: Generated new TSK 0
I20260810 07:24:47.282186 31051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:24:47.299211 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:47.294982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.24.60.254:33989)
0810 07:24:47.295043 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:47.295045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:47.295155 (+ 110us) server_negotiation.cc:407] Connection header received
0810 07:24:47.295213 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:47.295215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:47.295228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:47.295324 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:24:47.295441 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.296287 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.297295 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.297614 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.297694 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.297878 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:24:47.298957 (+ 1079us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 07:24:47.299109 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 07:24:47.299291 31040 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 07:24:47.301220 31066 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:47.294841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33989 (local address 127.0.0.1:53202)
0810 07:24:47.295081 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:47.295092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:47.295150 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:47.295329 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:47.295330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:47.295332 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:24:47.295387 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:47.295390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.296432 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.296433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:47.297138 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:47.297141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.297703 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.297704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:47.297752 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.297757 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:24:47.299070 (+ 1313us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 07:24:47.301087 (+ 2017us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.60.254:33989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1782,"thread_start_us":146,"threads_started":1}
I20260810 07:24:47.301411 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 30957
E20260810 07:24:47.313551 31068 webserver.cc:533] Webserver: sslize failed:
E20260810 07:24:47.313647 31068 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T07:24:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T07:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:47Z Disabled control of system clock
I20260810 07:24:47.352754 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36453
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:36453
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/testchainca.cert with env {}
W20260810 07:24:47.446134 31072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:47.446447 31072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:47.446498 31072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:47.447968 31072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:47.448040 31072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:47.448060 31072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:47.448082 31072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:47.448127 31072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:24:47.448155 31072 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:24:47.448184 31072 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 07:24:47.449995 31072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:36453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:47.450318 31072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:47.450621 31072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:47.456334 31079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:47.456336 31078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:47.456375 31081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:47.456353 31072 server_base.cc:1034] running on GCE node
I20260810 07:24:47.456920 31072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:47.457345 31072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:47.458539 31072 hybrid_clock.cc:648] HybridClock initialized: now 1786346687458513 us; error 37 us; skew 500 ppm
I20260810 07:24:47.460224 31072 webserver.cc:460] Webserver started at http://127.24.60.254:38803/ using document root <none> and password file <none>
I20260810 07:24:47.460580 31072 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:47.460664 31072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:47.460812 31072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:47.462071 31072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "bd8548022fb34769af3da8fc181e7b5e"
format_stamp: "Formatted at 2026-08-10 07:24:47 on dist-test-slave-vxj2"
server_key: "086aab186000ec0d055d33e5eb7f8f4c"
server_key_iv: "d131c20a637a6d3441289af9abcf765c"
server_key_version: "encryptionkey@0"
I20260810 07:24:47.462473 31072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "bd8548022fb34769af3da8fc181e7b5e"
format_stamp: "Formatted at 2026-08-10 07:24:47 on dist-test-slave-vxj2"
server_key: "086aab186000ec0d055d33e5eb7f8f4c"
server_key_iv: "d131c20a637a6d3441289af9abcf765c"
server_key_version: "encryptionkey@0"
I20260810 07:24:47.463985 31072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:24:47.464973 31089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:47.465243 31072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:47.465390 31072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "bd8548022fb34769af3da8fc181e7b5e"
format_stamp: "Formatted at 2026-08-10 07:24:47 on dist-test-slave-vxj2"
server_key: "086aab186000ec0d055d33e5eb7f8f4c"
server_key_iv: "d131c20a637a6d3441289af9abcf765c"
server_key_version: "encryptionkey@0"
I20260810 07:24:47.465469 31072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:47.476166 31072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:47.476567 31072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:47.476730 31072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:47.481102 31072 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:36453
I20260810 07:24:47.481210 31148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:36453 every 8 connection(s)
I20260810 07:24:47.481630 31072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:47.482841 31149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:47.485572 31149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e: Bootstrap starting.
I20260810 07:24:47.486192 31149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:47.486462 31149 log.cc:824] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:47.487289 31149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e: No bootstrap required, opened a new log
I20260810 07:24:47.488359 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31072
I20260810 07:24:47.488559 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:47.488771 24819 external_mini_cluster.cc:1428] Setting key 224081324a2ac6272f7719cfc155a566
I20260810 07:24:47.489486 31149 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8548022fb34769af3da8fc181e7b5e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36453 } }
I20260810 07:24:47.489836 31149 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:47.489960 31149 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8548022fb34769af3da8fc181e7b5e, State: Initialized, Role: FOLLOWER
I20260810 07:24:47.490163 31149 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8548022fb34769af3da8fc181e7b5e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36453 } }
I20260810 07:24:47.490295 31149 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:47.490355 31149 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:47.490419 31149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:47.491160 31149 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8548022fb34769af3da8fc181e7b5e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36453 } }
I20260810 07:24:47.491297 31149 leader_election.cc:302] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd8548022fb34769af3da8fc181e7b5e; no voters:
I20260810 07:24:47.491487 31149 leader_election.cc:288] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:47.491540 31155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:47.491731 31155 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [term 1 LEADER]: Becoming Leader. State: Replica: bd8548022fb34769af3da8fc181e7b5e, State: Running, Role: LEADER
I20260810 07:24:47.491868 31149 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:47.491895 31155 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8548022fb34769af3da8fc181e7b5e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36453 } }
I20260810 07:24:47.492254 31157 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd8548022fb34769af3da8fc181e7b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8548022fb34769af3da8fc181e7b5e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36453 } } }
I20260810 07:24:47.492352 31157 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:47.492244 31155 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8548022fb34769af3da8fc181e7b5e. Latest consensus state: current_term: 1 leader_uuid: "bd8548022fb34769af3da8fc181e7b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8548022fb34769af3da8fc181e7b5e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36453 } } }
I20260810 07:24:47.492566 31155 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:47.492691 31162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:47.493561 31162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:47.495731 31162 catalog_manager.cc:1347] Generated new cluster ID: 437b8572a2174cc8b71237c5900fb6be
I20260810 07:24:47.495790 31162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:47.496109 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:47.489696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.24.60.254:36453)
0810 07:24:47.489967 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:47.489976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:47.490021 (+ 45us) server_negotiation.cc:407] Connection header received
0810 07:24:47.490160 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:47.490167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:47.490235 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:47.490393 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:47.490658 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.492280 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.493560 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.494349 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.494503 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.495276 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:47.495280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:47.495280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:24:47.495513 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:47.495557 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:24:47.495572 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:47.495666 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:47.495797 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 07:24:47.495887 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20260810 07:24:47.497223 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:24:47.502148 31162 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:47.502720 31162 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:47.507409 31162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd8548022fb34769af3da8fc181e7b5e: Generated new TSK 0
I20260810 07:24:47.507668 31162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:24:47.539445 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:47.515347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.24.60.254:36453)
0810 07:24:47.515407 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:47.515409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:47.515501 (+ 92us) server_negotiation.cc:407] Connection header received
0810 07:24:47.515594 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:47.515595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:47.515607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:47.515681 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:24:47.515876 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.516369 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.517674 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.517948 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.518020 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.518290 (+ 270us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:24:47.539096 (+ 20806us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 07:24:47.539339 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42288: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 07:24:47.539543 31154 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42288: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 07:24:47.539943 31182 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:47.515193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36453 (local address 127.0.0.1:42288)
0810 07:24:47.515399 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:47.515413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:47.515503 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:47.515704 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:47.515705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:47.515708 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:24:47.515787 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:47.515790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.516503 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.516504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:47.517531 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:47.517536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:47.518051 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:47.518053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:47.518122 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:47.518130 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:24:47.539349 (+ 21219us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 07:24:47.539670 (+ 321us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.60.254:36453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":134,"threads_started":1}
I20260810 07:24:47.540445 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31072
2026-08-10T07:24:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (218 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (96 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:47 dist-test-slave-vxj2 krb5kdc[31195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:47 dist-test-slave-vxj2 krb5kdc[31195](info): set up 2 sockets
Aug 10 07:24:47 dist-test-slave-vxj2 krb5kdc[31195](info): commencing operation
krb5kdc: starting...
W20260810 07:24:48.867875 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:48 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:24:49.000414 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:38425
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:38425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:49.085886 31211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:49.086166 31211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:49.086210 31211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:49.087510 31211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:49.087590 31211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:49.087618 31211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:49.087641 31211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:49.087675 31211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:49.089358 31211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:38425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:38425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:49.089654 31211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:49.089901 31211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:49.095216 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.095239 31216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.095223 31219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:49.095597 31211 server_base.cc:1034] running on GCE node
I20260810 07:24:49.095938 31211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:49.096324 31211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:49.097500 31211 hybrid_clock.cc:648] HybridClock initialized: now 1786346689097489 us; error 49 us; skew 500 ppm
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.113135 31211 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:24:49.113762 31211 webserver.cc:460] Webserver started at http://127.24.60.254:43575/ using document root <none> and password file <none>
I20260810 07:24:49.113993 31211 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:49.114040 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:49.114145 31211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:49.115311 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "0f89966edee84445ad729493d7c37cd8"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "447fe9aca54d555e363416cb80931c82"
server_key_iv: "44e59169bc3128b4face51f54f6cb9c7"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.115660 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "0f89966edee84445ad729493d7c37cd8"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "447fe9aca54d555e363416cb80931c82"
server_key_iv: "44e59169bc3128b4face51f54f6cb9c7"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.117142 31211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 07:24:49.118067 31227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.118259 31211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:49.118352 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "0f89966edee84445ad729493d7c37cd8"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "447fe9aca54d555e363416cb80931c82"
server_key_iv: "44e59169bc3128b4face51f54f6cb9c7"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.118417 31211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:49.139077 31211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:49.141845 31211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:49.141995 31211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:49.146682 31211 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:38425
I20260810 07:24:49.146754 31291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:38425 every 8 connection(s)
I20260810 07:24:49.147140 31211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:49.148026 31293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:49.151253 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8: Bootstrap starting.
I20260810 07:24:49.152211 31293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:49.152616 31293 log.cc:824] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:49.153445 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8: No bootstrap required, opened a new log
I20260810 07:24:49.155122 31293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f89966edee84445ad729493d7c37cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 38425 } }
I20260810 07:24:49.155301 31293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:49.155352 31293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f89966edee84445ad729493d7c37cd8, State: Initialized, Role: FOLLOWER
I20260810 07:24:49.155512 31293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f89966edee84445ad729493d7c37cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 38425 } }
I20260810 07:24:49.155612 31293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:49.155658 31293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:49.155712 31293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:49.156293 31293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f89966edee84445ad729493d7c37cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 38425 } }
I20260810 07:24:49.156414 31293 leader_election.cc:302] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f89966edee84445ad729493d7c37cd8; no voters:
I20260810 07:24:49.156643 31293 leader_election.cc:288] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:49.156729 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31211
I20260810 07:24:49.156746 31297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:49.156850 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:49.156936 31293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:49.157051 24819 external_mini_cluster.cc:1428] Setting key 6e55c3868f677f741c1e3ce1aab936a8
I20260810 07:24:49.157189 31297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [term 1 LEADER]: Becoming Leader. State: Replica: 0f89966edee84445ad729493d7c37cd8, State: Running, Role: LEADER
I20260810 07:24:49.157369 31297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f89966edee84445ad729493d7c37cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 38425 } }
I20260810 07:24:49.157778 31297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f89966edee84445ad729493d7c37cd8. Latest consensus state: current_term: 1 leader_uuid: "0f89966edee84445ad729493d7c37cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f89966edee84445ad729493d7c37cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 38425 } } }
I20260810 07:24:49.157931 31299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f89966edee84445ad729493d7c37cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f89966edee84445ad729493d7c37cd8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 38425 } } }
I20260810 07:24:49.158013 31299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:49.157924 31297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:49.158377 31312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:49.159022 31312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:49.160578 31312 catalog_manager.cc:1347] Generated new cluster ID: 63d3bacb53aa4fdabd5e17beb1636f1d
I20260810 07:24:49.160635 31312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:49.170743 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.157751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.24.60.254:38425)
0810 07:24:49.157997 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.158007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.158046 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:24:49.158184 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.158191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.158255 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.158465 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.159291 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.160662 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.161831 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.162457 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.162907 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.165591 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.165596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.165597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.165612 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.168450 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.168920 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.168923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.168924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.169044 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.169299 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.169299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.169300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.169649 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.170286 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.170362 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.170515 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260810 07:24:49.179858 31312 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:49.180477 31312 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:49.188715 31312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f89966edee84445ad729493d7c37cd8: Generated new TSK 0
I20260810 07:24:49.189002 31312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:24:49.225929 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:38425
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:49.314018 31325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:49.314275 31325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:49.314316 31325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:49.315539 31325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:49.315618 31325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:49.315662 31325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:49.317267 31325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:49.317556 31325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:49.317801 31325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:49.323329 31335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.323320 31332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.323372 31331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:49.323673 31325 server_base.cc:1034] running on GCE node
I20260810 07:24:49.323917 31325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:49.324213 31325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:49.325385 31325 hybrid_clock.cc:648] HybridClock initialized: now 1786346689325365 us; error 35 us; skew 500 ppm
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.328198 31325 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:24:49.328701 31325 webserver.cc:460] Webserver started at http://127.24.60.193:44565/ using document root <none> and password file <none>
I20260810 07:24:49.328967 31325 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:49.329038 31325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:49.329233 31325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:49.330403 31325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "c337145abe3047bdb837e9b6c035b347"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "99c3edaee099112c8a0a2a205994096b"
server_key_iv: "77ae081b03f737ab6789425e31ed7803"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.330821 31325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "c337145abe3047bdb837e9b6c035b347"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "99c3edaee099112c8a0a2a205994096b"
server_key_iv: "77ae081b03f737ab6789425e31ed7803"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.332293 31325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:49.333240 31342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.333443 31325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:49.333559 31325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "c337145abe3047bdb837e9b6c035b347"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "99c3edaee099112c8a0a2a205994096b"
server_key_iv: "77ae081b03f737ab6789425e31ed7803"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.333643 31325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:49.344182 31325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:49.347173 31325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:49.347379 31325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:49.347643 31325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:49.348037 31325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:49.348097 31325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.348135 31325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:49.348163 31325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.354494 31325 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:33007
I20260810 07:24:49.354598 31477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:33007 every 8 connection(s)
I20260810 07:24:49.354995 31325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:49.363327 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31325
I20260810 07:24:49.363467 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:49.363754 24819 external_mini_cluster.cc:1428] Setting key b3e9c784cab33b06a020000a73be2341
I20260810 07:24:49.367213 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.356060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:36101 (local address 127.24.60.254:38425)
0810 07:24:49.356119 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.356122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.356303 (+ 181us) server_negotiation.cc:407] Connection header received
0810 07:24:49.357529 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.357531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.357550 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.357602 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.358406 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.358996 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.360028 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.360252 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.360325 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.363590 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.363593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.363593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.363602 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.365608 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.366124 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.366125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.366126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.366193 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.366475 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.366477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.366477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.366614 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.366688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.367019 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.367117 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 07:24:49.367442 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.355951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:38425 (local address 127.24.60.193:36101)
0810 07:24:49.356155 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:49.356210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:49.357394 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:49.357681 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:49.357684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:49.358062 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:49.358288 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:49.358294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.359089 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.359090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:49.359931 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:49.359933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.360353 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.360355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:49.360485 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.361227 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:49.361237 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:49.363402 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:49.365718 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:49.365733 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:49.365736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:49.366031 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:49.366267 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:49.366268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:49.366268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:49.366410 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:49.366691 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:49.366700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:49.366869 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 07:24:49.366983 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 07:24:49.367170 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260810 07:24:49.368367 31479 heartbeater.cc:347] Connected to a master server at 127.24.60.254:38425
I20260810 07:24:49.368505 31479 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:49.368794 31479 heartbeater.cc:511] Master 127.24.60.254:38425 requested a full tablet report, sending...
I20260810 07:24:49.369576 31250 ts_manager.cc:194] Registered new tserver with Master: c337145abe3047bdb837e9b6c035b347 (127.24.60.193:33007)
I20260810 07:24:49.370563 31250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:36101
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:24:49.404282 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:38425
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:49.506901 31485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:49.507140 31485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:49.507167 31485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:49.508349 31485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:49.508394 31485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:49.508420 31485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:49.510078 31485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:49.510296 31485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:49.510499 31485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:49.515897 31492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.515897 31491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.515897 31495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:49.516186 31485 server_base.cc:1034] running on GCE node
I20260810 07:24:49.516434 31485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:49.516690 31485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:49.517850 31485 hybrid_clock.cc:648] HybridClock initialized: now 1786346689517835 us; error 46 us; skew 500 ppm
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.520678 31485 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:24:49.521232 31485 webserver.cc:460] Webserver started at http://127.24.60.194:41799/ using document root <none> and password file <none>
I20260810 07:24:49.521497 31485 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:49.521546 31485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:49.521646 31485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:49.522742 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "b2e2943bdc1d4127929feae143b16c07"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "a5bfbdcd83626dc55c5b47d6b7d948e9"
server_key_iv: "d92c724eab21e04c1ba75b1b8f42d424"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.523061 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "b2e2943bdc1d4127929feae143b16c07"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "a5bfbdcd83626dc55c5b47d6b7d948e9"
server_key_iv: "d92c724eab21e04c1ba75b1b8f42d424"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.524451 31485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:24:49.525303 31502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.525561 31485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:49.525661 31485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "b2e2943bdc1d4127929feae143b16c07"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "a5bfbdcd83626dc55c5b47d6b7d948e9"
server_key_iv: "d92c724eab21e04c1ba75b1b8f42d424"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.525727 31485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:49.540482 31485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:49.543105 31485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:49.543239 31485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:49.543453 31485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:49.543817 31485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:49.543856 31485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.543880 31485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:49.543923 31485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.550628 31485 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:33363
I20260810 07:24:49.550698 31629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:33363 every 8 connection(s)
I20260810 07:24:49.551155 31485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:49.551483 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31485
I20260810 07:24:49.551599 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:49.551847 24819 external_mini_cluster.cc:1428] Setting key 8f9597e7a94847ef76716dfc9df362c3
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
I20260810 07:24:49.564311 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.552021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:33341 (local address 127.24.60.254:38425)
0810 07:24:49.552073 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.552076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.552515 (+ 439us) server_negotiation.cc:407] Connection header received
0810 07:24:49.553985 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.553986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.553999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.554061 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.554952 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.555731 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.557497 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.557739 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.557915 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.560823 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.560825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.560825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.560833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.562589 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.563093 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.563095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.563095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.563165 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.563473 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.563475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.563475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.563609 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.563692 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.564099 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.564206 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:24:49.564507 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.552055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:38425 (local address 127.24.60.194:33341)
0810 07:24:49.552357 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:49.552404 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:49.553841 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:49.554164 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:49.554169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:49.554615 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:49.554831 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:49.554836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.555847 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.555848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:49.557351 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:49.557354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.557926 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.557927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:49.558101 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.558900 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:49.558910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:49.560689 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:49.562706 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:49.562716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:49.562719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:49.563001 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:49.563242 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:49.563243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:49.563243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:49.563386 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:49.563681 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:49.563690 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:49.563897 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 07:24:49.564022 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 07:24:49.564223 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":227,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
I20260810 07:24:49.565253 31630 heartbeater.cc:347] Connected to a master server at 127.24.60.254:38425
I20260810 07:24:49.565392 31630 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:49.565629 31630 heartbeater.cc:511] Master 127.24.60.254:38425 requested a full tablet report, sending...
I20260810 07:24:49.566159 31250 ts_manager.cc:194] Registered new tserver with Master: b2e2943bdc1d4127929feae143b16c07 (127.24.60.194:33363)
I20260810 07:24:49.566615 31250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:33341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:24:49.585755 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:38425
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:49.672222 31635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:49.672470 31635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:49.672505 31635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:49.673866 31635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:49.673924 31635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:49.673954 31635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:49.675572 31635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:49.675854 31635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:49.676096 31635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:49.682153 31642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:49.682178 31644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:49.682497 31635 server_base.cc:1034] running on GCE node
W20260810 07:24:49.682165 31641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:49.682880 31635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:49.683143 31635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:49.684299 31635 hybrid_clock.cc:648] HybridClock initialized: now 1786346689684289 us; error 37 us; skew 500 ppm
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.687525 31635 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:24:49.688077 31635 webserver.cc:460] Webserver started at http://127.24.60.195:39111/ using document root <none> and password file <none>
I20260810 07:24:49.688346 31635 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:49.688417 31635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:49.688570 31635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:49.689826 31635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "8c642da6eed949efb5aa5f75c8827caf"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "097e0e062fcff6abe806573e8fbc181a"
server_key_iv: "e99a8b9faa13e1f2121744453a9cf8c6"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.690229 31635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "8c642da6eed949efb5aa5f75c8827caf"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "097e0e062fcff6abe806573e8fbc181a"
server_key_iv: "e99a8b9faa13e1f2121744453a9cf8c6"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.691944 31635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:49.692934 31654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.693238 31635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:49.693351 31635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "8c642da6eed949efb5aa5f75c8827caf"
format_stamp: "Formatted at 2026-08-10 07:24:49 on dist-test-slave-vxj2"
server_key: "097e0e062fcff6abe806573e8fbc181a"
server_key_iv: "e99a8b9faa13e1f2121744453a9cf8c6"
server_key_version: "encryptionkey@0"
I20260810 07:24:49.693439 31635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:49.721745 31635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:49.724109 31635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:49.724294 31635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:49.724562 31635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:49.724931 31635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:49.724987 31635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.725031 31635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:49.725061 31635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:49.732142 31635 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:45879
I20260810 07:24:49.732189 31785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:45879 every 8 connection(s)
I20260810 07:24:49.732564 31635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:24:49.742596 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31635
I20260810 07:24:49.742713 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:49.742913 24819 external_mini_cluster.cc:1428] Setting key 2354242c05e5dc81c22c7d14a5963230
I20260810 07:24:49.744305 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.733436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:37949 (local address 127.24.60.254:38425)
0810 07:24:49.733525 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.733528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.733891 (+ 363us) server_negotiation.cc:407] Connection header received
0810 07:24:49.734939 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.734941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.734953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.735016 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.735765 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.736337 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.737582 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.737786 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.737910 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.740671 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.740673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.740673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.740687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.742327 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.742931 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.742932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.742933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.743025 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.743384 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.743386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.743386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.743551 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.743629 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.744070 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.744198 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:24:49.744536 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.733413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:38425 (local address 127.24.60.195:37949)
0810 07:24:49.733654 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:49.733698 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:49.734770 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:49.735074 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:49.735077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:49.735438 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:49.735655 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:49.735661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.736461 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.736462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:49.737495 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:49.737499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.737920 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.737921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:49.738082 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.738629 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:49.738638 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:49.740554 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:49.742472 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:49.742481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:49.742483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:49.742761 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:49.743105 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:49.743106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:49.743106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:49.743279 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:49.743636 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:49.743645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:49.743861 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 07:24:49.743999 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 07:24:49.744237 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":130,"threads_started":1}
I20260810 07:24:49.745191 31786 heartbeater.cc:347] Connected to a master server at 127.24.60.254:38425
I20260810 07:24:49.745333 31786 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:49.745537 31786 heartbeater.cc:511] Master 127.24.60.254:38425 requested a full tablet report, sending...
I20260810 07:24:49.745955 31250 ts_manager.cc:194] Registered new tserver with Master: 8c642da6eed949efb5aa5f75c8827caf (127.24.60.195:45879)
I20260810 07:24:49.746380 31250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:37949
I20260810 07:24:49.755357 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/test-admin.keytab.
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.775588 24819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 07:24:49.777663 31794 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:49.776653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.24.60.193:33007)
0810 07:24:49.776900 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.776904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.776916 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:24:49.776946 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.776948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.776962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.777037 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.777542 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46194: BlockingRecv error: recv got EOF from 127.0.0.1:46194 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":122,"threads_started":1}
W20260810 07:24:49.777696 24819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc' of type FILE
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.778919 24819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.780079 24819 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:49.789529 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.780705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.24.60.193:33007)
0810 07:24:49.780805 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.780807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.780818 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:49.780885 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.780886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.780903 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.780953 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.781522 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.782288 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.783092 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.783485 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.783581 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.786273 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.786285 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.786285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.786299 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.788208 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.788631 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.788635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.788635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.788693 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.788899 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.788900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.788900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.789049 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.789256 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.789316 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.789411 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 07:24:49.791690 31794 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:49.791063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.24.60.193:33007)
0810 07:24:49.791149 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.791152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.791163 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:49.791215 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.791216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.791228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.791280 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.791621 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46204: BlockingRecv error: recv got EOF from 127.0.0.1:46204 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 07:24:49.791718 24819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc' of type FILE
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.792794 24819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.793782 24819 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:49.801416 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.794345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.24.60.193:33007)
0810 07:24:49.794399 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.794401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.794424 (+ 23us) server_negotiation.cc:407] Connection header received
0810 07:24:49.794479 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.794479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.794489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.794536 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.794898 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.795389 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.796149 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.796357 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.796413 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.798706 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.798707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.798708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.798715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.800121 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.800496 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.800497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.800498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.800563 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.800810 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.800810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.800811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.800914 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.801011 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.801268 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.801349 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 07:24:49.803161 31794 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:49.802651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.24.60.193:33007)
0810 07:24:49.802709 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.802711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.802720 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:49.802766 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.802767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.802777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.802820 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.803110 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46222: BlockingRecv error: recv got EOF from 127.0.0.1:46222 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 07:24:49.803371 24819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346663269703-24819-0/krb5kdc/krb5cc' of type FILE
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.804524 24819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:49.805527 24819 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.193@KRBTEST.COM
I20260810 07:24:49.813723 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:49.806182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.24.60.193:33007)
0810 07:24:49.806245 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:49.806247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:49.806264 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:24:49.806286 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:49.806287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:49.806294 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:49.806345 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:49.806771 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.807360 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.808085 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:49.808289 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:49.808382 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:49.810813 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:49.810815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:49.810815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:49.810826 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:49.812391 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.812788 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.812789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.812790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.812860 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:49.813050 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:49.813051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:49.813051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:49.813229 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:49.813314 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:49.813469 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 07:24:49.813599 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:24:49.814716 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31325
I20260810 07:24:49.822422 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31485
I20260810 07:24:49.831516 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31635
I20260810 07:24:49.839956 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31211
2026-08-10T07:24:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2108 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31808](info): set up 2 sockets
Aug 10 07:24:49 dist-test-slave-vxj2 krb5kdc[31808](info): commencing operation
krb5kdc: starting...
W20260810 07:24:51.032265 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab.
Entry for principal oryx/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab.
I20260810 07:24:51.169737 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45641
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab
--principal=oryx/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:45641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:51.256563 31824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:51.256827 31824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:51.256855 31824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:51.258101 31824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:51.258149 31824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:51.258164 31824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:51.258177 31824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:51.258190 31824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:51.259720 31824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:45641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab
--principal=oryx/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:51.259958 31824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:51.260166 31824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:51.265550 31831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:51.265703 31830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:51.265770 31824 server_base.cc:1034] running on GCE node
W20260810 07:24:51.265554 31833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:51.266209 31824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:51.266599 31824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:51.267792 31824 hybrid_clock.cc:648] HybridClock initialized: now 1786346691267739 us; error 77 us; skew 500 ppm
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:51.271109 31824 init.cc:377] Logged in from keytab as oryx/127.24.60.254@KRBTEST.COM (short username oryx)
I20260810 07:24:51.271682 31824 webserver.cc:460] Webserver started at http://127.24.60.254:44859/ using document root <none> and password file <none>
I20260810 07:24:51.271960 31824 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:51.272033 31824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:51.272207 31824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:51.273451 31824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "0ac336ad4dd04ec5bbb549375c5f006b"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "aa95aca9507df0bbf1404cea9917e743"
server_key_iv: "24694f368f8c8daaa957235f269d48a7"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.273861 31824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "0ac336ad4dd04ec5bbb549375c5f006b"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "aa95aca9507df0bbf1404cea9917e743"
server_key_iv: "24694f368f8c8daaa957235f269d48a7"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.275395 31824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:51.276352 31840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.276557 31824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:51.276674 31824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "0ac336ad4dd04ec5bbb549375c5f006b"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "aa95aca9507df0bbf1404cea9917e743"
server_key_iv: "24694f368f8c8daaa957235f269d48a7"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.276764 31824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:51.313640 31824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:51.316039 31824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:51.316224 31824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:51.320785 31824 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:45641
I20260810 07:24:51.320822 31903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:45641 every 8 connection(s)
I20260810 07:24:51.321250 31824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:51.322361 31904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:51.324827 31904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Bootstrap starting.
I20260810 07:24:51.325445 31904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:51.325700 31904 log.cc:824] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:51.326085 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31824
I20260810 07:24:51.326228 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:51.326434 24819 external_mini_cluster.cc:1428] Setting key 80bf86837a57da91db6a66c0b33dcd69
I20260810 07:24:51.326556 31904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: No bootstrap required, opened a new log
I20260810 07:24:51.328665 31904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } }
I20260810 07:24:51.328869 31904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:51.328940 31904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac336ad4dd04ec5bbb549375c5f006b, State: Initialized, Role: FOLLOWER
I20260810 07:24:51.329090 31904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } }
I20260810 07:24:51.329221 31904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:51.329280 31904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:51.329366 31904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:51.330394 31904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } }
I20260810 07:24:51.330521 31904 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac336ad4dd04ec5bbb549375c5f006b; no voters:
I20260810 07:24:51.330726 31904 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:51.330751 31910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:51.330875 31910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 LEADER]: Becoming Leader. State: Replica: 0ac336ad4dd04ec5bbb549375c5f006b, State: Running, Role: LEADER
I20260810 07:24:51.331024 31904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:51.331027 31910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } }
I20260810 07:24:51.331529 31912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac336ad4dd04ec5bbb549375c5f006b. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } }
I20260810 07:24:51.331625 31912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:51.331831 31911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } }
I20260810 07:24:51.331960 31911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
W20260810 07:24:51.333413 31931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:24:51.333477 31931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:24:51.333528 31921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:51.333715 31921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:51.335592 31921 catalog_manager.cc:1347] Generated new cluster ID: 3666b311175d451e981f62823ebc0070
I20260810 07:24:51.335646 31921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:51.339570 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.327241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.24.60.254:45641)
0810 07:24:51.327488 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:51.327497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:51.327526 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:24:51.328666 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:51.328676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:51.328740 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:51.328980 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:51.329834 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.330919 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.331947 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.332480 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.332602 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.335529 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:51.335534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:51.335535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:51.335549 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:51.337713 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.338180 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.338183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.338183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.338318 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.338544 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.338545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.338545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.338872 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:51.339021 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:51.339216 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:24:51.339364 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":133,"threads_started":1}
I20260810 07:24:51.342043 31921 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:51.342636 31921 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:51.346396 31921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Generated new TSK 0
I20260810 07:24:51.346638 31921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab.
Entry for principal oryx/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab.
I20260810 07:24:51.382094 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45641
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:51.474134 31937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:51.474359 31937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:51.474383 31937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:51.475507 31937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:51.475549 31937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:51.475576 31937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:51.477095 31937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:51.477387 31937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:51.477661 31937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:51.483219 31943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:51.483372 31937 server_base.cc:1034] running on GCE node
W20260810 07:24:51.483229 31946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:51.483469 31942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:51.483705 31937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:51.483986 31937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:51.485219 31937 hybrid_clock.cc:648] HybridClock initialized: now 1786346691485263 us; error 89 us; skew 500 ppm
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:51.488215 31937 init.cc:377] Logged in from keytab as oryx/127.24.60.193@KRBTEST.COM (short username oryx)
I20260810 07:24:51.488704 31937 webserver.cc:460] Webserver started at http://127.24.60.193:44637/ using document root <none> and password file <none>
I20260810 07:24:51.488960 31937 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:51.489030 31937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:51.489230 31937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:51.490486 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "6d6593755a904596852db7c05c2b91a4"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "ac2b6b1a131e138357b3dbc2b0670596"
server_key_iv: "80d9571060b775a03b0dc8720c9d31dc"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.490885 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "6d6593755a904596852db7c05c2b91a4"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "ac2b6b1a131e138357b3dbc2b0670596"
server_key_iv: "80d9571060b775a03b0dc8720c9d31dc"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.492384 31937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:51.493327 31956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.493549 31937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:51.493662 31937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "6d6593755a904596852db7c05c2b91a4"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "ac2b6b1a131e138357b3dbc2b0670596"
server_key_iv: "80d9571060b775a03b0dc8720c9d31dc"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.493744 31937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:51.512130 31937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:51.514807 31937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:51.515003 31937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:51.515285 31937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:51.515654 31937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:51.515710 31937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.515753 31937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:51.515789 31937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.521871 31937 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36675
I20260810 07:24:51.521944 32088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36675 every 8 connection(s)
I20260810 07:24:51.522271 31937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:51.532047 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 31937
I20260810 07:24:51.532188 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:51.532480 24819 external_mini_cluster.cc:1428] Setting key 86014130393439a97d99f1e89a4d2fbc
I20260810 07:24:51.533579 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.523276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:33347 (local address 127.24.60.254:45641)
0810 07:24:51.523379 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:51.523381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:51.523598 (+ 217us) server_negotiation.cc:407] Connection header received
0810 07:24:51.524130 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:51.524131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:51.524146 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:51.524215 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:51.525353 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.525920 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.527097 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.527330 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.527397 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.530023 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:51.530025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:51.530025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:51.530033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:51.531603 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.532150 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.532152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.532152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.532225 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.532616 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.532618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.532619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.532827 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:51.532907 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:51.533410 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 07:24:51.533503 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 07:24:51.533879 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.523238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:33347)
0810 07:24:51.523467 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:51.523512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:51.524032 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:51.524296 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:51.524299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:51.524977 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:51.525215 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:51.525228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.526047 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.526048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:51.527000 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:51.527003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.527419 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.527420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:51.527540 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.528156 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:51.528170 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:51.529905 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:51.531694 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:51.531702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:51.531705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:51.532059 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:51.532323 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:51.532324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:51.532324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:51.532503 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:51.532947 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:51.532955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:51.533226 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 07:24:51.533351 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 07:24:51.533565 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":141,"threads_started":1}
I20260810 07:24:51.535269 32089 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45641
I20260810 07:24:51.535391 32089 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:51.535627 32089 heartbeater.cc:511] Master 127.24.60.254:45641 requested a full tablet report, sending...
I20260810 07:24:51.536295 31861 ts_manager.cc:194] Registered new tserver with Master: 6d6593755a904596852db7c05c2b91a4 (127.24.60.193:36675)
I20260810 07:24:51.536964 31861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:33347
WARNING: no policy specified for oryx/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab.
Entry for principal oryx/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab.
I20260810 07:24:51.561908 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45641
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:51.646978 32095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:51.647243 32095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:51.647270 32095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:51.648491 32095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:51.648547 32095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:51.648578 32095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:51.650174 32095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:51.650429 32095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:51.650632 32095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:51.656553 32103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:51.656540 32101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:51.656546 32100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:51.656579 32095 server_base.cc:1034] running on GCE node
I20260810 07:24:51.657192 32095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:51.657483 32095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:51.658679 32095 hybrid_clock.cc:648] HybridClock initialized: now 1786346691658668 us; error 37 us; skew 500 ppm
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:51.661756 32095 init.cc:377] Logged in from keytab as oryx/127.24.60.194@KRBTEST.COM (short username oryx)
I20260810 07:24:51.662328 32095 webserver.cc:460] Webserver started at http://127.24.60.194:44199/ using document root <none> and password file <none>
I20260810 07:24:51.662595 32095 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:51.662662 32095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:51.662809 32095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:51.663915 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "457f8ebb4e7cb1d91b87d37ef06a6a69"
server_key_iv: "79ce8d1842e8071a47bfac7f064d7d13"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.664291 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "457f8ebb4e7cb1d91b87d37ef06a6a69"
server_key_iv: "79ce8d1842e8071a47bfac7f064d7d13"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.665916 32095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:51.666891 32110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.667141 32095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:51.667258 32095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "457f8ebb4e7cb1d91b87d37ef06a6a69"
server_key_iv: "79ce8d1842e8071a47bfac7f064d7d13"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.667404 32095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:51.692181 32095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:51.694682 32095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:51.694871 32095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:51.695156 32095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:51.695509 32095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:51.695564 32095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.695609 32095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:51.695636 32095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.702163 32095 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:40999
I20260810 07:24:51.702256 32243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:40999 every 8 connection(s)
I20260810 07:24:51.702620 32095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:51.708226 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 32095
I20260810 07:24:51.708345 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:51.708563 24819 external_mini_cluster.cc:1428] Setting key 6f55a49164569bf331adf954da404043
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:51.713680 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.703518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:55979 (local address 127.24.60.254:45641)
0810 07:24:51.703612 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:51.703615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:51.703821 (+ 206us) server_negotiation.cc:407] Connection header received
0810 07:24:51.704284 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:51.704285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:51.704298 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:51.704381 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:51.705457 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.706063 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.707218 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.707499 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.707587 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.710462 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:51.710464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:51.710464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:51.710474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:51.711934 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.712511 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.712512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.712513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.712581 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.712901 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.712902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.712902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.713030 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:51.713093 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:51.713522 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 07:24:51.713609 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:24:51.714174 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.703431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:55979)
0810 07:24:51.703626 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:51.703670 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:51.704178 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:51.704439 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:51.704443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:51.705076 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:51.705321 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:51.705328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.706182 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.706183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:51.707100 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:51.707103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.707621 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.707623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:51.707759 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.708589 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:51.708601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:51.710347 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:51.712054 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:51.712065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:51.712067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:51.712423 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:51.712657 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:51.712658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:51.712658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:51.712822 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:51.713103 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:51.713110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:51.713336 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 07:24:51.713477 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 07:24:51.713689 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260810 07:24:51.714893 32244 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45641
I20260810 07:24:51.715023 32244 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:51.715257 32244 heartbeater.cc:511] Master 127.24.60.254:45641 requested a full tablet report, sending...
I20260810 07:24:51.715706 31861 ts_manager.cc:194] Registered new tserver with Master: 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999)
I20260810 07:24:51.716114 31861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:55979
WARNING: no policy specified for oryx/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab.
Entry for principal oryx/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab.
I20260810 07:24:51.740716 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45641
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:51.830924 32251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:51.831281 32251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:51.831359 32251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:51.832870 32251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:51.832924 32251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:51.832957 32251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:51.834666 32251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:51.834977 32251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:51.835225 32251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:51.840682 32261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:51.840696 32258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:51.840682 32259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:51.841187 32251 server_base.cc:1034] running on GCE node
I20260810 07:24:51.841444 32251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:51.841707 32251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:51.842859 32251 hybrid_clock.cc:648] HybridClock initialized: now 1786346691842863 us; error 45 us; skew 500 ppm
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:51.845768 32251 init.cc:377] Logged in from keytab as oryx/127.24.60.195@KRBTEST.COM (short username oryx)
I20260810 07:24:51.846347 32251 webserver.cc:460] Webserver started at http://127.24.60.195:44351/ using document root <none> and password file <none>
I20260810 07:24:51.846647 32251 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:51.846720 32251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:51.846896 32251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:51.848091 32251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "48ef028d25ec4faeb78bd84a23eac893"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "c58829613ff4ddd7ad2754c76534ccc8"
server_key_iv: "921752ceffda934d7f07ca33cc5877a4"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.848480 32251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "48ef028d25ec4faeb78bd84a23eac893"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "c58829613ff4ddd7ad2754c76534ccc8"
server_key_iv: "921752ceffda934d7f07ca33cc5877a4"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.850229 32251 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 07:24:51.851207 32270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.851485 32251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:51.851630 32251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "48ef028d25ec4faeb78bd84a23eac893"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "c58829613ff4ddd7ad2754c76534ccc8"
server_key_iv: "921752ceffda934d7f07ca33cc5877a4"
server_key_version: "encryptionkey@0"
I20260810 07:24:51.851722 32251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:51.863144 32251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:51.866032 32251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:51.866241 32251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:51.866549 32251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:51.866933 32251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:51.866991 32251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.867038 32251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:51.867087 32251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:51.874133 32251 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:41733
I20260810 07:24:51.874219 32398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:41733 every 8 connection(s)
I20260810 07:24:51.874594 32251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:51.877210 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 32251
I20260810 07:24:51.877298 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:51.877583 24819 external_mini_cluster.cc:1428] Setting key efa2034b15def7fd870d7eed4f1ee6e2
Aug 10 07:24:51 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:51.886713 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.875600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:43947 (local address 127.24.60.254:45641)
0810 07:24:51.875665 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:51.875667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:51.875740 (+ 73us) server_negotiation.cc:407] Connection header received
0810 07:24:51.876529 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:51.876531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:51.876542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:51.876602 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:51.877888 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.878769 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.879984 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.880214 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.880279 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.883361 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:51.883362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:51.883363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:51.883371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:51.885032 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.885634 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.885636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.885636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.885698 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:51.886048 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:51.886049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:51.886049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:51.886188 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:51.886259 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:51.886560 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 07:24:51.886641 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:24:51.887024 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:51.875411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:43947)
0810 07:24:51.875586 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:51.875632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:51.876389 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:51.876667 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:51.876671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:51.877510 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:51.877774 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:51.877787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.878871 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.878872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:51.879829 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:51.879833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:51.880328 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:51.880329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:51.880452 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:51.881120 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:51.881130 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:51.883060 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:51.885228 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:51.885238 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:51.885240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:51.885540 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:51.885783 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:51.885784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:51.885784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:51.885938 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:51.886262 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:51.886268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:51.886419 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 07:24:51.886503 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:24:51.886669 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260810 07:24:51.887708 32399 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45641
I20260810 07:24:51.887873 32399 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:51.888089 32399 heartbeater.cc:511] Master 127.24.60.254:45641 requested a full tablet report, sending...
I20260810 07:24:51.888511 31861 ts_manager.cc:194] Registered new tserver with Master: 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733)
I20260810 07:24:51.888887 31861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:43947
I20260810 07:24:51.889837 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.60.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.253.keytab.
Entry for principal oryx/127.24.60.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.253.keytab.
I20260810 07:24:51.920369 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:35835
--webserver_interface=127.24.60.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.253.keytab
--principal=oryx/127.24.60.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:45641,127.24.60.253:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:52.007082 32405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:52.007352 32405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:52.007393 32405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:52.008628 32405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:52.008700 32405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:52.008723 32405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:52.008744 32405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:52.008770 32405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:52.010525 32405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:45641,127.24.60.253:35835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.253.keytab
--principal=oryx/127.24.60.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.60.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
W20260810 07:24:52.010703 32405 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.60.254:45641,127.24.60.253:35835'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 07:24:52.010869 32405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:52.011116 32405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.016609 32410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.016633 32411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.016582 32413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.016908 32405 server_base.cc:1034] running on GCE node
I20260810 07:24:52.017239 32405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.017573 32405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.018736 32405 hybrid_clock.cc:648] HybridClock initialized: now 1786346692018690 us; error 56 us; skew 500 ppm
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.021833 32405 init.cc:377] Logged in from keytab as oryx/127.24.60.253@KRBTEST.COM (short username oryx)
I20260810 07:24:52.022449 32405 webserver.cc:460] Webserver started at http://127.24.60.253:41207/ using document root <none> and password file <none>
I20260810 07:24:52.022683 32405 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.022749 32405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.022928 32405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:52.024085 32405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/instance:
uuid: "2cf16959c76b4f5b8e70446d3a07e900"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "d9bdd42787eea0f9f1ea7457f0dd637d"
server_key_iv: "a01f40dfbe05deb105c188050625afbe"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.024458 32405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal/instance:
uuid: "2cf16959c76b4f5b8e70446d3a07e900"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "d9bdd42787eea0f9f1ea7457f0dd637d"
server_key_iv: "a01f40dfbe05deb105c188050625afbe"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.026029 32405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:52.027006 32424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.027246 32405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:52.027360 32405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
uuid: "2cf16959c76b4f5b8e70446d3a07e900"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "d9bdd42787eea0f9f1ea7457f0dd637d"
server_key_iv: "a01f40dfbe05deb105c188050625afbe"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.027431 32405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.057827 32405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.060278 32405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:52.060407 32405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.064747 32405 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.253:35835
I20260810 07:24:52.064884 32479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.253:35835 every 8 connection(s)
I20260810 07:24:52.065237 32405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
I20260810 07:24:52.066579 32480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:52.066582 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 32405
I20260810 07:24:52.068925 32480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.253@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:52.079221 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.069679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.24.60.254:45641)
0810 07:24:52.069771 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.069773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.069916 (+ 143us) server_negotiation.cc:407] Connection header received
0810 07:24:52.070004 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.070005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.070019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.070076 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.070659 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.071444 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.072559 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.072872 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.072957 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.075695 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.075697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.075698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.075707 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.077699 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.078192 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.078193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.078193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.078274 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.078472 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.078473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.078473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.078680 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.078749 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.079012 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.079116 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:24:52.079464 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.067334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.24.60.253:35835)
0810 07:24:52.067601 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.067611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.067646 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:24:52.068121 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.068128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.068176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.068331 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.068982 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.070391 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.071525 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.072159 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.072268 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.075081 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.075091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.075091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.075116 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.077370 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.077775 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.077779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.077780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.077896 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.078191 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.078192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.078193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.078607 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.078779 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.078986 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.079198 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":137,"threads_started":1}
I20260810 07:24:52.079489 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.069624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:39712)
0810 07:24:52.069804 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.069845 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.069918 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.070106 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.070109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.070500 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.070572 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.070574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.071543 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.071543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.072449 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.072452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.072985 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.072986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.073061 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.073661 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.073669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.075580 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.077808 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.077816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.077818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.078096 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.078350 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.078350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.078351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.078406 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.078748 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.078755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.078833 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.078987 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.079310 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":120,"thread_start_us":106,"threads_started":1}
I20260810 07:24:52.080379 32480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.253@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
I20260810 07:24:52.088271 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.080687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:34390)
0810 07:24:52.080745 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.080758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.080826 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.081019 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.081020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.081260 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.081347 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.081350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.082082 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.082083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.082659 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.082660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.083083 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.083084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.083158 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.083678 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.083685 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.085235 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.087114 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.087115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.087115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.087301 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.087523 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.087524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.087525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.087586 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.087858 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.087858 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.087904 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.087999 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.088134 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":5}
I20260810 07:24:52.088318 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.080782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.24.60.253:35835)
0810 07:24:52.080908 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.080909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.080922 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:24:52.080940 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.080940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.080948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.081016 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.081433 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.081995 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.082771 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.082997 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.083080 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.085362 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.085363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.085364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.085372 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.086981 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.087385 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.087387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.087387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.087449 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.087658 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.087659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.087660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.087788 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.087868 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.088022 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.088202 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":106}
I20260810 07:24:52.090121 32480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Bootstrap starting.
I20260810 07:24:52.091099 32480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:52.091434 32480 log.cc:824] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:52.092351 32480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: No bootstrap required, opened a new log
I20260810 07:24:52.093969 32480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } }
I20260810 07:24:52.094115 32480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:52.094139 32480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf16959c76b4f5b8e70446d3a07e900, State: Initialized, Role: FOLLOWER
I20260810 07:24:52.094295 32480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } }
I20260810 07:24:52.094640 32480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:52.094619 32489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } } }
I20260810 07:24:52.094769 32489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:24:52.096655 32502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:24:52.096725 32502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:24:52.098466 32405 master_runner.cc:428] Detected that this master 2cf16959c76b4f5b8e70446d3a07e900 is joining an existing cluster
I20260810 07:24:52.098526 32405 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.60.253:35835
I20260810 07:24:52.107067 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.099941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.24.60.253:35835)
0810 07:24:52.099982 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.099984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.100209 (+ 225us) server_negotiation.cc:407] Connection header received
0810 07:24:52.100274 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.100275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.100284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.100339 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.100712 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.101506 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.102325 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.102521 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.102605 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.103883 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.103884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.103884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.103890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.105638 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.106132 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.106134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.106134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.106199 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.106420 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.106421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.106422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.106565 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.106649 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.106818 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.106933 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":23}
I20260810 07:24:52.107410 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.099909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:34402)
0810 07:24:52.100146 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.100155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.100218 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.100355 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.100355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.100541 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.100626 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.100628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.101596 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.101597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.102243 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.102245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.102616 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.102617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.102680 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.103215 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.103221 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.103789 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.105760 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.105760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.105761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.106029 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.106275 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.106276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.106276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.106354 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.106639 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.106640 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.106686 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.106906 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.107289 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260810 07:24:52.108072 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.099690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.24.60.254:45641)
0810 07:24:52.099755 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.099758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.099977 (+ 219us) server_negotiation.cc:407] Connection header received
0810 07:24:52.100010 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.100011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.100023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.100080 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.100874 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.101718 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.102831 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.103047 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.103129 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.104365 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.104367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.104367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.104376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.106638 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.107092 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.107094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.107094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.107166 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.107417 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.107417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.107418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.107563 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.107610 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.107875 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.107983 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 07:24:52.108398 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.099637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:39728)
0810 07:24:52.099795 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.099808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.099881 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.100103 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.100104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.100703 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.100778 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.100781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.101801 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.101803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.102731 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.102734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.103198 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.103199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.103264 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.103678 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.103683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.104155 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.106723 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.106724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.106725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.107012 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.107253 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.107254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.107254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.107322 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.107665 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.107667 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.107734 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.108170 (+ 436us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.108285 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.254@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
I20260810 07:24:52.119748 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.111410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.24.60.253:35835)
0810 07:24:52.111459 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.111461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.111582 (+ 121us) server_negotiation.cc:407] Connection header received
0810 07:24:52.111708 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.111709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.111720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.111775 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.112336 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.112834 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.113699 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.113913 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.113990 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.116558 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.116560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.116560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.116569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.118213 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.118777 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.118779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.118779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.118907 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.119155 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.119156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.119156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.119290 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.119361 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.119577 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.119661 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 07:24:52.119800 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.111274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:34404)
0810 07:24:52.111453 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.111515 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.111613 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.111780 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.111784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.112164 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.112226 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.112229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.112918 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.112920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.113611 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.113614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.113991 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.113992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.114075 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.114670 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.114675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.116447 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.118317 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.118339 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.118343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.118673 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.118991 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.118992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.118992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.119072 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.119369 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.119378 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.119446 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.119544 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.119693 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260810 07:24:52.120563 31861 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.60.253:35835: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900"
member_type: NON_VOTER
last_known_addr {
host: "127.24.60.253"
port: 35835
}
attrs {
promote: true
}
}
dest_uuid: "0ac336ad4dd04ec5bbb549375c5f006b"
cas_config_opid_index: -1
I20260810 07:24:52.120944 31861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } }
I20260810 07:24:52.121572 32514 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2cf16959c76b4f5b8e70446d3a07e900 (127.24.60.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } }
I20260810 07:24:52.121987 32454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:52.122972 32514 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.60.253:35835
I20260810 07:24:52.123167 32454 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Refusing update from remote peer 0ac336ad4dd04ec5bbb549375c5f006b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 07:24:52.123189 32514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } }
I20260810 07:24:52.123270 32514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.123265 32405 master.cc:537] Master@127.24.60.253:35835 shutting down...
I20260810 07:24:52.123636 32514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:24:52.124543 31845 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 07:24:52.126165 31845 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b -> Peer 2cf16959c76b4f5b8e70446d3a07e900 (127.24.60.253:35835): Couldn't send request to peer 2cf16959c76b4f5b8e70446d3a07e900. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 07:24:52.126328 32405 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:52.126432 32405 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:52.126489 32405 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: stopping tablet replica
I20260810 07:24:52.129344 32405 master.cc:552] Master@127.24.60.253:35835 shutdown complete.
I20260810 07:24:52.129423 32405 master_runner.cc:305] Clearing existing system tablet
I20260810 07:24:52.129578 32405 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.129644 32405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.130367 32405 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:52.131201 32519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.131378 32405 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.131443 32405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:24:52.131494 32405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
uuid: "2cf16959c76b4f5b8e70446d3a07e900"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "d9bdd42787eea0f9f1ea7457f0dd637d"
server_key_iv: "a01f40dfbe05deb105c188050625afbe"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.132454 32405 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:52.134354 32405 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 07:24:52.134434 32405 log.cc:1196] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 07:24:52.134666 32405 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Deleting consensus metadata
I20260810 07:24:52.134802 32405 master_runner.cc:315] Copying system tablet from 127.24.60.254:45641
I20260810 07:24:52.135511 32405 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: tablet copy: Beginning tablet copy session from remote peer at address 127.24.60.254:45641
I20260810 07:24:52.143725 32525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.135886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:39734)
0810 07:24:52.136107 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.136118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.136192 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.136350 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.136351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.136550 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.136596 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.136597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.137643 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.137644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.138248 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.138250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.138768 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.138769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.138828 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.139871 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.139876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.140237 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.142373 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.142374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.142375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.142593 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.142923 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.142924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.142925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.143007 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.143344 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.143346 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.143421 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.143494 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.143621 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260810 07:24:52.143739 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.136023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.24.60.254:45641)
0810 07:24:52.136100 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.136102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.136188 (+ 86us) server_negotiation.cc:407] Connection header received
0810 07:24:52.136246 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.136247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.136260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.136327 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.136671 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.137508 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.138354 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.138676 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.138755 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.140365 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.140368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.140368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.140378 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.142244 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.142679 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.142680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.142681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.142755 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.143084 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.143085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.143086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.143256 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.143341 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.143536 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.143636 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:24:52.144313 31885 tablet_copy_service.cc:140] P 0ac336ad4dd04ec5bbb549375c5f006b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2cf16959c76b4f5b8e70446d3a07e900 ({username='oryx', principal='oryx/127.24.60.253@KRBTEST.COM'} at 127.0.0.1:39734)
I20260810 07:24:52.144455 31885 tablet_copy_service.cc:161] P 0ac336ad4dd04ec5bbb549375c5f006b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2cf16959c76b4f5b8e70446d3a07e900 at {username='oryx', principal='oryx/127.24.60.253@KRBTEST.COM'} at 127.0.0.1:39734: session id = 2cf16959c76b4f5b8e70446d3a07e900-00000000000000000000000000000000
I20260810 07:24:52.145424 31885 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 07:24:52.146205 32405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 07:24:52.147413 32405 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: tablet copy: Starting download of 0 data blocks...
I20260810 07:24:52.147593 32405 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: tablet copy: Starting download of 1 WAL segments...
I20260810 07:24:52.148614 32405 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 07:24:52.149744 31885 tablet_copy_service.cc:342] P 0ac336ad4dd04ec5bbb549375c5f006b: Request end of tablet copy session 2cf16959c76b4f5b8e70446d3a07e900-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.60.253@KRBTEST.COM'} at 127.0.0.1:39734
I20260810 07:24:52.149822 31885 tablet_copy_service.cc:433] P 0ac336ad4dd04ec5bbb549375c5f006b: ending tablet copy session 2cf16959c76b4f5b8e70446d3a07e900-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2cf16959c76b4f5b8e70446d3a07e900
I20260810 07:24:52.152609 32405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.154012 32526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.154099 32530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.154290 32528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.154315 32405 server_base.cc:1034] running on GCE node
I20260810 07:24:52.154543 32405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.154718 32405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.155932 32405 hybrid_clock.cc:648] HybridClock initialized: now 1786346692155910 us; error 39 us; skew 500 ppm
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.157557 32405 init.cc:377] Logged in from keytab as oryx/127.24.60.253@KRBTEST.COM (short username oryx)
I20260810 07:24:52.157887 32405 webserver.cc:460] Webserver started at http://127.24.60.253:40295/ using document root <none> and password file <none>
I20260810 07:24:52.158023 32405 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.158069 32405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.158794 32405 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:52.159394 32537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.159519 32405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.159569 32405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
uuid: "2cf16959c76b4f5b8e70446d3a07e900"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "d9bdd42787eea0f9f1ea7457f0dd637d"
server_key_iv: "a01f40dfbe05deb105c188050625afbe"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.159638 32405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.170300 32405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.172034 32405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.175375 32405 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.253:35835
I20260810 07:24:52.175737 32598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.253:35835 every 8 connection(s)
I20260810 07:24:52.175817 32405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
I20260810 07:24:52.176801 32601 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:24:52.177299 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Bootstrap starting.
I20260810 07:24:52.179872 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:24:52.180130 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Bootstrap complete.
I20260810 07:24:52.180371 32601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } }
I20260810 07:24:52.180475 32601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2cf16959c76b4f5b8e70446d3a07e900, State: Initialized, Role: LEARNER
I20260810 07:24:52.180531 32601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } }
I20260810 07:24:52.180750 32601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:52.180744 32604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } }
I20260810 07:24:52.180956 32604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: LEARNER
I20260810 07:24:52.182389 32619 catalog_manager.cc:1259] Loaded cluster ID: 3666b311175d451e981f62823ebc0070
I20260810 07:24:52.182442 32619 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: loading cluster ID for follower catalog manager: success
I20260810 07:24:52.183756 32619 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: acquiring CA information for follower catalog manager: success
I20260810 07:24:52.184106 32619 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:24:52.190371 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.181489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.24.60.253:35835)
0810 07:24:52.182583 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.182585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.182597 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:24:52.182623 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.182624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.182635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.183586 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.183791 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.184658 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.185588 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.185811 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.185893 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.187304 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.187306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.187306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.187314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.189013 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.189477 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.189478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.189478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.189547 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.189758 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.189759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.189759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.189893 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.189997 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.190125 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.190228 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for oryx/127.24.60.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.252.keytab.
Entry for principal oryx/127.24.60.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.252.keytab.
I20260810 07:24:52.220722 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:38349
--webserver_interface=127.24.60.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.252.keytab
--principal=oryx/127.24.60.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:52.314004 32622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:52.314257 32622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:52.314282 32622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:52.315640 32622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:52.315702 32622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:52.315724 32622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:52.315743 32622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:52.315761 32622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:52.317977 32622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.252.keytab
--principal=oryx/127.24.60.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.60.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:52.318286 32622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:52.318547 32622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.323868 32628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.324043 32631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.324136 32629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.324247 32622 server_base.cc:1034] running on GCE node
I20260810 07:24:52.324527 32622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.324831 32622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.325989 32622 hybrid_clock.cc:648] HybridClock initialized: now 1786346692325976 us; error 29 us; skew 500 ppm
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.328897 32622 init.cc:377] Logged in from keytab as oryx/127.24.60.252@KRBTEST.COM (short username oryx)
I20260810 07:24:52.329478 32622 webserver.cc:460] Webserver started at http://127.24.60.252:46259/ using document root <none> and password file <none>
I20260810 07:24:52.329691 32622 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.329734 32622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.329834 32622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:52.331048 32622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/instance:
uuid: "1da4db7cdd3741d3b5468b919147bed1"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "f617f8474ff55010f9e4afafd7e896b1"
server_key_iv: "9d82d20fdb91bc0b6d2ab9915fc54b86"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.331388 32622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal/instance:
uuid: "1da4db7cdd3741d3b5468b919147bed1"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "f617f8474ff55010f9e4afafd7e896b1"
server_key_iv: "9d82d20fdb91bc0b6d2ab9915fc54b86"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.333262 32622 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:24:52.334172 32639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.334340 32622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:52.334483 32622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
uuid: "1da4db7cdd3741d3b5468b919147bed1"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "f617f8474ff55010f9e4afafd7e896b1"
server_key_iv: "9d82d20fdb91bc0b6d2ab9915fc54b86"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.334570 32622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.344555 32622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.347573 32622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:52.347787 32622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.352906 32622 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.252:38349
I20260810 07:24:52.352959 32703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.252:38349 every 8 connection(s)
I20260810 07:24:52.353447 32622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
I20260810 07:24:52.354557 32704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:52.356371 32704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:24:52.356935 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 32622
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.252@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.60.252@KRBTEST.COM
I20260810 07:24:52.367755 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.357359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.24.60.254:45641)
0810 07:24:52.357441 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.357444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.357698 (+ 254us) server_negotiation.cc:407] Connection header received
0810 07:24:52.358344 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.358346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.358362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.358440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.359093 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.359721 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.360804 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.361015 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.361107 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.364086 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.364088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.364088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.364098 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.366030 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.366527 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.366529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.366529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.366599 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.366937 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.366937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.366938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.367068 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.367122 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.367515 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.367632 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:24:52.367877 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.357154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:39742)
0810 07:24:52.357541 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.357609 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.358196 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.358466 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.358470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.358797 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.358982 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.358987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.359837 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.359838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.360685 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.360688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.361178 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.361180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.361349 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.361964 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.361974 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.363899 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.366143 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.366152 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.366154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.366430 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.366694 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.366695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.366696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.366833 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.367138 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.367147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.367336 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.367467 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.367672 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":29,"thread_start_us":153,"threads_started":1}
I20260810 07:24:52.368309 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.357652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.24.60.252:38349)
0810 07:24:52.357802 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.357806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.357848 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:24:52.358219 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.358237 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.358273 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.358406 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.359057 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.360007 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.360849 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.361303 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.361425 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.364638 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.364653 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.364653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.364660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.366831 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.367324 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.367327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.367327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.367389 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.367652 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.367653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.367653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.367847 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.368024 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.368111 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.368200 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260810 07:24:52.368976 32704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.252@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
I20260810 07:24:52.377115 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.369285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:34418)
0810 07:24:52.369357 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.369367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.369420 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.369595 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.369596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.369772 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.369813 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.369815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.370526 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.370527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.371214 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.371224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.371735 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.371736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.371820 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.372310 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.372314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.373773 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.375905 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.375906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.375907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.376102 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.376352 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.376353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.376353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.376420 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.376729 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.376730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.376787 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.376880 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.377009 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 07:24:52.377113 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.369371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.24.60.253:35835)
0810 07:24:52.369439 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.369442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.369451 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:52.369478 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.369479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.369494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.369591 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.369924 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.370432 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.371346 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.371629 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.371733 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.373966 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.373968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.373969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.373979 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.375727 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.376196 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.376204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.376204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.376266 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.376494 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.376495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.376495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.376657 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.376740 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.376913 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.377016 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:24:52.377751 32704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.252@KRBTEST.COM for oryx/127.24.60.252@KRBTEST.COM
I20260810 07:24:52.387581 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.378092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33072)
0810 07:24:52.378146 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.378161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.378209 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.378385 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.378386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.378530 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.378578 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.378580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.379304 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.379306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.380101 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.380104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.380674 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.380675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.380747 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.381529 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.381535 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.383507 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.386053 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.386055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.386056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.386337 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.386682 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.386683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.386684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.386784 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.387167 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.387169 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.387228 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.387314 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.387454 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 07:24:52.387708 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.378128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.24.60.252:38349)
0810 07:24:52.378172 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.378174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.378195 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:24:52.378260 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.378261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.378269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.378317 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.378652 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.379198 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.380198 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.380553 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.380622 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.383646 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.383649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.383649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.383672 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.385801 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.386464 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.386465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.386466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.386571 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.386873 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.386874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.386874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.387064 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.387160 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.387369 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.387567 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":20}
I20260810 07:24:52.390385 32704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Bootstrap starting.
I20260810 07:24:52.391624 32704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:52.392033 32704 log.cc:824] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:52.393205 32704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: No bootstrap required, opened a new log
I20260810 07:24:52.397969 32704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } }
I20260810 07:24:52.398227 32704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:52.398273 32704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da4db7cdd3741d3b5468b919147bed1, State: Initialized, Role: FOLLOWER
I20260810 07:24:52.398437 32704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } }
I20260810 07:24:52.398875 32704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:52.398854 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } } }
I20260810 07:24:52.399019 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:24:52.401115 32728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:24:52.401289 32728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:24:52.403858 32622 master_runner.cc:428] Detected that this master 1da4db7cdd3741d3b5468b919147bed1 is joining an existing cluster
I20260810 07:24:52.403923 32622 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.60.252:38349
I20260810 07:24:52.413980 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.405276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.24.60.254:45641)
0810 07:24:52.405341 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.405343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.405452 (+ 109us) server_negotiation.cc:407] Connection header received
0810 07:24:52.405550 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.405551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.405566 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.405653 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.406131 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.406962 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408071 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.408375 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408414 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.409817 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.409819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.409819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.409828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.412274 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.412871 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.412873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.412873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.412958 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.413250 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.413251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.413252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.413416 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.413505 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.413768 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.413869 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 07:24:52.413988 32735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.405121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:39746)
0810 07:24:52.405369 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.405382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.405472 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.405653 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.405654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.405910 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.406014 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.406017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.407058 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.407058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.407970 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.407972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.408522 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.408586 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.409142 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.409147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.409701 (+ 554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.412417 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.412418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.412419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.412766 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.413049 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.413050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.413051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.413129 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.413536 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.413537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.413618 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.413714 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.413863 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":176,"threads_started":1}
I20260810 07:24:52.420984 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31937
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346691, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.254@KRBTEST.COM for oryx/127.24.60.252@KRBTEST.COM
I20260810 07:24:52.435617 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:36675
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:45641
--webserver_port=44637
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:24:52.438939 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.416721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33082)
0810 07:24:52.416797 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.416813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.416885 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.417185 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.417186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.417406 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.417465 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.417467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.418249 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.418251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.418854 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.418857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.419311 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.419313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.419405 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.419840 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.419845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.424249 (+ 4404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.437647 (+ 13398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.437649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.437649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.437907 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.438142 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.438143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.438143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.438204 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.438591 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.438592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.438648 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.438698 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.438826 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 07:24:52.439325 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.416786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.24.60.252:38349)
0810 07:24:52.416996 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.416998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.417006 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:52.417053 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.417054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.417063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.417146 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.417555 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.418164 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.418950 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.419213 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.424400 (+ 5187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.424518 (+ 118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.424519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.424519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.424527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.437513 (+ 12986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.438003 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.438004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.438005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.438067 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.438272 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.438273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.438273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.438469 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.439081 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.439128 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.439221 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":122,"threads_started":1}
I20260810 07:24:52.439981 31861 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.60.252:38349: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1"
member_type: NON_VOTER
last_known_addr {
host: "127.24.60.252"
port: 38349
}
attrs {
promote: true
}
}
dest_uuid: "0ac336ad4dd04ec5bbb549375c5f006b"
cas_config_opid_index: 5
I20260810 07:24:52.440279 31861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } }
I20260810 07:24:52.441022 32512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.441847 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.405879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.24.60.253:35835)
0810 07:24:52.405930 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.405931 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.405941 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:52.406016 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.406017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.406029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.406096 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.406638 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.407370 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408463 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.408759 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408812 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.437453 (+ 28641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.437456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.437456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.437467 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.439395 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.440850 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.440852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.440852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.440928 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.441204 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.441205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.441206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.441368 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.441421 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.441647 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.441743 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:52.442248 32512 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.60.252:38349
I20260810 07:24:52.442332 32512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.442389 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.405462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:34432)
0810 07:24:52.405884 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.405895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.405954 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.406155 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.406156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.406434 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.406521 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.406524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.407490 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.407491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.408339 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.408342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.408905 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.408979 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.436153 (+ 27174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.436161 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.437282 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.439900 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.439901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.439901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.440474 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.441026 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.441026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.441027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.441093 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.441478 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.441478 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.441536 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.441993 (+ 457us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.442267 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":81,"threads_started":1}
I20260810 07:24:52.442401 32512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.442641 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.405763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.24.60.252:38349)
0810 07:24:52.405807 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.405809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.406110 (+ 301us) server_negotiation.cc:407] Connection header received
0810 07:24:52.406189 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.406189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.406197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.406222 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.406643 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.407302 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408250 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.408516 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408553 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.439716 (+ 31163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.439718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.439718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.439725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.441432 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.441799 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.441800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.441801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.441843 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.442011 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.442011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.442011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.442152 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.442232 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.442477 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.442555 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:52.442816 32622 master.cc:537] Master@127.24.60.252:38349 shutting down...
I20260810 07:24:52.443375 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.405689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33074)
0810 07:24:52.406057 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.406066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.406128 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.406267 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.406267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.406491 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.406563 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.406565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.407380 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.407381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.408175 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.408177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.408617 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.408618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.408675 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.436307 (+ 27632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.436314 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.439618 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.441534 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.441535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.441536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.441729 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.441903 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.441903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.441903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.441948 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.442220 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.442221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.442333 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.442708 (+ 375us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.442801 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":83,"threads_started":1}
W20260810 07:24:52.445263 31844 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b -> Peer 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252:38349): Couldn't send request to peer 1da4db7cdd3741d3b5468b919147bed1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 07:24:52.448045 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.440803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:34434)
0810 07:24:52.443947 (+ 3144us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.443960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.444029 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.444344 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.444345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.444348 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:52.444413 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.444416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.445394 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.445395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.446675 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.446679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.447741 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.447742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.447818 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.447851 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.447913 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.447962 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3123}
I20260810 07:24:52.448122 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.440907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.24.60.253:35835)
0810 07:24:52.441936 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.441938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.444041 (+ 2103us) server_negotiation.cc:407] Connection header received
0810 07:24:52.444074 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.444075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.444104 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.444189 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:52.444507 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.445293 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.446869 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.447538 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.447611 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.447699 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.447947 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.448009 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":70,"threads_started":1}
I20260810 07:24:52.448354 32622 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:52.448467 32622 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:52.448542 32622 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: stopping tablet replica
I20260810 07:24:52.448576 32572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 LEARNER]: Refusing update from remote peer 0ac336ad4dd04ec5bbb549375c5f006b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 07:24:52.448884 32514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 07:24:52.449659 32572 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.451165 32604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac336ad4dd04ec5bbb549375c5f006b. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.451314 32604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: LEARNER
I20260810 07:24:52.451373 32604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: NON_VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: true } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.451452 32604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: LEARNER
I20260810 07:24:52.451678 32622 master.cc:552] Master@127.24.60.252:38349 shutdown complete.
I20260810 07:24:52.451733 32622 master_runner.cc:305] Clearing existing system tablet
I20260810 07:24:52.451880 32622 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.451933 32622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.452848 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.452316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33094)
0810 07:24:52.452764 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.452802 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406}
I20260810 07:24:52.452813 32514 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: attempting to promote NON_VOTER 2cf16959c76b4f5b8e70446d3a07e900 to VOTER
I20260810 07:24:52.453118 32514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } }
I20260810 07:24:52.453778 32572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 LEARNER]: Refusing update from remote peer 0ac336ad4dd04ec5bbb549375c5f006b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 07:24:52.453999 32512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 07:24:52.454157 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.453985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33106)
0810 07:24:52.454055 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.454111 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 07:24:52.454710 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.454584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33112)
0810 07:24:52.454628 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.454667 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 07:24:52.454819 31844 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b -> Peer 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252:38349): Couldn't send request to peer 1da4db7cdd3741d3b5468b919147bed1. Status: Network error: Client connection negotiation failed: client connection to 127.24.60.252:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 07:24:52.455369 32622 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.000s
I20260810 07:24:52.455336 32512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2cf16959c76b4f5b8e70446d3a07e900 (127.24.60.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.456442 32512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.456521 32512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.456816 32514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.456940 32514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.456990 32572 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2cf16959c76b4f5b8e70446d3a07e900 (127.24.60.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.458427 32743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.458549 32743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:24:52.461299 32747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.461473 32622 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.461526 32622 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260810 07:24:52.461550 32622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
uuid: "1da4db7cdd3741d3b5468b919147bed1"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "f617f8474ff55010f9e4afafd7e896b1"
server_key_iv: "9d82d20fdb91bc0b6d2ab9915fc54b86"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.462452 32622 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:52.464530 32622 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 07:24:52.464624 32622 log.cc:1196] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 07:24:52.464852 32622 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Deleting consensus metadata
I20260810 07:24:52.464978 32622 master_runner.cc:315] Copying system tablet from 127.24.60.254:45641
I20260810 07:24:52.465760 32622 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.60.254:45641
I20260810 07:24:52.474371 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.466227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.24.60.254:45641)
0810 07:24:52.466318 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.466321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.466472 (+ 151us) server_negotiation.cc:407] Connection header received
0810 07:24:52.466566 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.466567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.466581 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.466665 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.467156 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.467950 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.469298 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.469628 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.469708 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.471124 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.471126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.471126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.471135 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.472956 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.473476 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.473478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.473478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.473544 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.473773 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.473774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.473774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.473920 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.473992 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.474179 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.474270 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:24:52.474395 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.466142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:39756)
0810 07:24:52.466380 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.466392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.466467 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.466686 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.466688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.466937 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.467041 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.467046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.468058 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.468059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.469122 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.469127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.469789 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.469791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.469868 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.470457 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.470463 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.471032 (+ 569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.473060 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.473061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.473062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.473385 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.473620 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.473621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.473621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.473701 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.473992 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.473993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.474055 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.474139 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.474270 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260810 07:24:52.475198 31885 tablet_copy_service.cc:140] P 0ac336ad4dd04ec5bbb549375c5f006b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1da4db7cdd3741d3b5468b919147bed1 ({username='oryx', principal='oryx/127.24.60.252@KRBTEST.COM'} at 127.0.0.1:39756)
I20260810 07:24:52.475271 31885 tablet_copy_service.cc:161] P 0ac336ad4dd04ec5bbb549375c5f006b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1da4db7cdd3741d3b5468b919147bed1 at {username='oryx', principal='oryx/127.24.60.252@KRBTEST.COM'} at 127.0.0.1:39756: session id = 1da4db7cdd3741d3b5468b919147bed1-00000000000000000000000000000000
I20260810 07:24:52.475724 31885 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 07:24:52.476368 32622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 07:24:52.478019 32622 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: tablet copy: Starting download of 0 data blocks...
I20260810 07:24:52.478178 32622 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: tablet copy: Starting download of 1 WAL segments...
I20260810 07:24:52.479223 32622 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 07:24:52.480381 31885 tablet_copy_service.cc:342] P 0ac336ad4dd04ec5bbb549375c5f006b: Request end of tablet copy session 1da4db7cdd3741d3b5468b919147bed1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.60.252@KRBTEST.COM'} at 127.0.0.1:39756
I20260810 07:24:52.480443 31885 tablet_copy_service.cc:433] P 0ac336ad4dd04ec5bbb549375c5f006b: ending tablet copy session 1da4db7cdd3741d3b5468b919147bed1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1da4db7cdd3741d3b5468b919147bed1
I20260810 07:24:52.483467 32622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.484969 32755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.489975 32756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.490288 32758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.490442 32622 server_base.cc:1034] running on GCE node
I20260810 07:24:52.490653 32622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.490896 32622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.492105 32622 hybrid_clock.cc:648] HybridClock initialized: now 1786346692492091 us; error 30 us; skew 500 ppm
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.494164 32622 init.cc:377] Logged in from keytab as oryx/127.24.60.252@KRBTEST.COM (short username oryx)
I20260810 07:24:52.494558 32622 webserver.cc:460] Webserver started at http://127.24.60.252:35033/ using document root <none> and password file <none>
I20260810 07:24:52.494753 32622 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.494885 32622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.495786 32622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:52.496543 32767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.496719 32622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 07:24:52.496831 32622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
uuid: "1da4db7cdd3741d3b5468b919147bed1"
format_stamp: "Formatted at 2026-08-10 07:24:52 on dist-test-slave-vxj2"
server_key: "f617f8474ff55010f9e4afafd7e896b1"
server_key_iv: "9d82d20fdb91bc0b6d2ab9915fc54b86"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.496966 32622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.512406 32622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.514647 32622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.518033 32622 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.252:38349
I20260810 07:24:52.518083 367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.252:38349 every 8 connection(s)
I20260810 07:24:52.518595 32622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
I20260810 07:24:52.519742 368 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:24:52.520334 368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Bootstrap starting.
I20260810 07:24:52.523849 368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:24:52.524132 368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: Bootstrap complete.
I20260810 07:24:52.524367 368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } }
I20260810 07:24:52.524487 368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1da4db7cdd3741d3b5468b919147bed1, State: Initialized, Role: LEARNER
I20260810 07:24:52.524533 368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } }
I20260810 07:24:52.524736 368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:52.524756 371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.524984 371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: This master's current role is: LEARNER
I20260810 07:24:52.526448 384 catalog_manager.cc:1259] Loaded cluster ID: 3666b311175d451e981f62823ebc0070
I20260810 07:24:52.526551 384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: loading cluster ID for follower catalog manager: success
I20260810 07:24:52.527809 384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: acquiring CA information for follower catalog manager: success
I20260810 07:24:52.528195 384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 07:24:52.562011 32740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:52.562362 32740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:52.562414 32740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:52.564342 32740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:52.564430 32740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:52.564483 32740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:52.566486 32740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:36675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=44637
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:52.566851 32740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:52.567137 32740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.573800 391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.573840 392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.573793 395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.574345 32740 server_base.cc:1034] running on GCE node
I20260810 07:24:52.574579 32740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.574867 32740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.576042 32740 hybrid_clock.cc:648] HybridClock initialized: now 1786346692576022 us; error 34 us; skew 500 ppm
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.579617 32740 init.cc:377] Logged in from keytab as oryx/127.24.60.193@KRBTEST.COM (short username oryx)
I20260810 07:24:52.580389 32740 webserver.cc:460] Webserver started at http://127.24.60.193:44637/ using document root <none> and password file <none>
I20260810 07:24:52.580705 32740 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.580802 32740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.582686 32740 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:52.583563 403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.583734 32740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:52.583837 32740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "6d6593755a904596852db7c05c2b91a4"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "ac2b6b1a131e138357b3dbc2b0670596"
server_key_iv: "80d9571060b775a03b0dc8720c9d31dc"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.584205 32740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.595264 32740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.597944 32740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:52.598137 32740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.598431 32740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:52.598783 32740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:52.598847 32740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.598899 32740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:52.598928 32740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.605420 32740 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36675
I20260810 07:24:52.605485 538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36675 every 8 connection(s)
I20260810 07:24:52.605877 32740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:52.610962 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 32740
I20260810 07:24:52.611156 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 32095
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.252@KRBTEST.COM
I20260810 07:24:52.619433 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.606707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:39693 (local address 127.24.60.254:45641)
0810 07:24:52.606767 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.606769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.607250 (+ 481us) server_negotiation.cc:407] Connection header received
0810 07:24:52.608287 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.608289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.608302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.608335 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.609122 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.609942 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611132 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.611440 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611485 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.615605 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.615607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.615607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.615616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.617549 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.618180 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.618181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.618182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.618243 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.618576 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.618577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.618577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.618709 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.618760 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.619243 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.619335 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 07:24:52.619655 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.606710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.24.60.193:36923)
0810 07:24:52.606963 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.607001 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.608101 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.608411 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.608416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.608768 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.608993 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.608999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.609933 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.609934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.610907 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.610910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.611664 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.611772 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.613021 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.613027 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.615825 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.617987 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.617988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.617988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.618202 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.618435 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.618436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.618436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.618486 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.618803 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.618812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.619042 (+ 230us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.619196 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.619397 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260810 07:24:52.619798 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.606710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:39693)
0810 07:24:52.607173 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.607186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.608144 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.608429 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.608429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.608814 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.608995 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.609002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.610050 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.610051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.611035 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.611036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.611592 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.611767 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.612708 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.612713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.615501 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.617667 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.617683 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.617687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.618086 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.618321 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.618322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.618322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.618477 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.618823 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.618823 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.619042 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.619576 (+ 534us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.619681 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":95,"thread_start_us":103,"threads_started":1}
I20260810 07:24:52.620131 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.606708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:36923 (local address 127.24.60.252:38349)
0810 07:24:52.606836 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.606838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.607191 (+ 353us) server_negotiation.cc:407] Connection header received
0810 07:24:52.608224 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.608226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.608242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.608318 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.609100 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.609821 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611021 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.611533 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611603 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.615938 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.615939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.615940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.615947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.617891 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.618299 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.618300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.618301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.618364 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.618553 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.618554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.618554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.618710 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.618774 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.619915 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.620021 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260810 07:24:52.620393 540 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45641
I20260810 07:24:52.620504 540 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.620726 540 heartbeater.cc:511] Master 127.24.60.254:45641 requested a full tablet report, sending...
I20260810 07:24:52.621214 31861 ts_manager.cc:194] Re-registered known tserver with Master: 6d6593755a904596852db7c05c2b91a4 (127.24.60.193:36675)
I20260810 07:24:52.621376 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.607150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:41525 (local address 127.24.60.253:35835)
0810 07:24:52.607215 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.607217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.607339 (+ 122us) server_negotiation.cc:407] Connection header received
0810 07:24:52.608275 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.608277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.608294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.608330 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.609122 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.609941 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611131 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.611440 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611485 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.615357 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.615359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.615359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.615371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.618895 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.619448 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.619450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.619451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.619531 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.620647 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.620648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.620648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.620820 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.620898 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.621186 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.621296 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 07:24:52.621505 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.607055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.24.60.193:41525)
0810 07:24:52.607277 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.607285 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.608144 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.608426 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.608427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.608861 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.608995 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.609002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.610050 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.610051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.611020 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.611023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.611591 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.611592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.611767 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.612682 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.612694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.615173 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.619032 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.619033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.619034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.619335 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.620463 (+ 1128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.620464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.620465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.620558 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.620915 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.620916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.620985 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.621123 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.621306 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":71,"thread_start_us":59,"threads_started":1}
I20260810 07:24:52.621755 31861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:39693
I20260810 07:24:52.627992 539 heartbeater.cc:347] Connected to a master server at 127.24.60.252:38349
I20260810 07:24:52.628108 539 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.628257 539 heartbeater.cc:511] Master 127.24.60.252:38349 requested a full tablet report, sending...
I20260810 07:24:52.628831 321 ts_manager.cc:194] Registered new tserver with Master: 6d6593755a904596852db7c05c2b91a4 (127.24.60.193:36675)
I20260810 07:24:52.630702 541 heartbeater.cc:347] Connected to a master server at 127.24.60.253:35835
I20260810 07:24:52.630788 541 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.630872 541 heartbeater.cc:511] Master 127.24.60.253:35835 requested a full tablet report, sending...
I20260810 07:24:52.631707 32559 ts_manager.cc:194] Registered new tserver with Master: 6d6593755a904596852db7c05c2b91a4 (127.24.60.193:36675)
I20260810 07:24:52.639394 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:40999
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:45641
--webserver_port=44199
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:52.727326 547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:52.727586 547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:52.727629 547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:52.728854 547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:52.728929 547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:52.728976 547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:52.730592 547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:40999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=44199
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:52.730906 547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:52.731208 547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.736984 558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.736974 555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.737040 547 server_base.cc:1034] running on GCE node
W20260810 07:24:52.736974 556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.737421 547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.737687 547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.738847 547 hybrid_clock.cc:648] HybridClock initialized: now 1786346692738823 us; error 41 us; skew 500 ppm
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.741693 547 init.cc:377] Logged in from keytab as oryx/127.24.60.194@KRBTEST.COM (short username oryx)
I20260810 07:24:52.742178 547 webserver.cc:460] Webserver started at http://127.24.60.194:44199/ using document root <none> and password file <none>
I20260810 07:24:52.742379 547 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.742435 547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.743794 547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:52.744438 566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.744661 547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:52.744761 547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "457f8ebb4e7cb1d91b87d37ef06a6a69"
server_key_iv: "79ce8d1842e8071a47bfac7f064d7d13"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.745129 547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.765923 547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.768621 547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:52.768778 547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.769093 547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:52.769479 547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:52.769515 547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.769557 547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:52.769582 547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.775619 547 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:40999
I20260810 07:24:52.775709 699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:40999 every 8 connection(s)
I20260810 07:24:52.776078 547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.252@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
I20260810 07:24:52.785424 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 547
I20260810 07:24:52.785624 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 32251
I20260810 07:24:52.789841 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.776988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.24.60.194:36775)
0810 07:24:52.777209 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.777262 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.778007 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.778287 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.778291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.778707 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.778938 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.778944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.779747 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.779749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.780939 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.780942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.781464 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.781632 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.782426 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.782439 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.785330 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.787948 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.787959 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.787962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.788222 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.788467 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.788468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.788468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.788615 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.788912 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.788921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.789095 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.789251 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.789482 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260810 07:24:52.791515 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.777506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:51937 (local address 127.24.60.254:45641)
0810 07:24:52.777646 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.777648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.777657 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:52.778136 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.778138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.778151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.778184 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.779069 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.779844 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781048 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.781361 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781399 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.784909 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.784917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.784918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.784929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.790212 (+ 5283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.790665 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.790667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.790667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.790729 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.790914 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.790915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.790915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.791063 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.791104 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.791288 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.791383 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 07:24:52.791918 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.777324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:51937)
0810 07:24:52.777453 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.777465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.778017 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.778724 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.778724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.778871 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.778940 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.778946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.779951 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.779952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.780940 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.780943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.781500 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.781632 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.782426 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.782439 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.784691 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.790328 (+ 5637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.790329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.790330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.790560 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.790801 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.790802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.790802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.790855 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.791146 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.791147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.791189 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.791639 (+ 450us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.791764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":31,"thread_start_us":72,"threads_started":1}
I20260810 07:24:52.792673 701 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45641
I20260810 07:24:52.792827 701 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.793187 701 heartbeater.cc:511] Master 127.24.60.254:45641 requested a full tablet report, sending...
I20260810 07:24:52.793294 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.777581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:43513 (local address 127.24.60.252:38349)
0810 07:24:52.777631 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.777634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.777781 (+ 147us) server_negotiation.cc:407] Connection header received
0810 07:24:52.778135 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.778136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.778153 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.778184 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.779070 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.779877 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781121 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.781473 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.784452 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.784454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.784454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.784465 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.792007 (+ 7542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.792418 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.792419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.792420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.792478 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.792659 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.792660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.792660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.792803 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.792842 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.793024 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.793122 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:24:52.793659 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.777382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.24.60.194:43513)
0810 07:24:52.777704 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.777713 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.778020 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.778310 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.778311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.778722 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.778941 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.778946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.779987 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.779988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.781000 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.781002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.781549 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.781679 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.782460 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.782466 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.784297 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.792117 (+ 7820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.792118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.792119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.792322 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.792544 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.792544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.792545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.792598 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.792884 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.792885 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.792926 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.793374 (+ 448us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.793498 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":59,"thread_start_us":88,"threads_started":1}
I20260810 07:24:52.793853 31861 ts_manager.cc:194] Re-registered known tserver with Master: 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999)
I20260810 07:24:52.794154 700 heartbeater.cc:347] Connected to a master server at 127.24.60.252:38349
I20260810 07:24:52.794219 700 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.794312 31861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:51937
I20260810 07:24:52.794318 700 heartbeater.cc:511] Master 127.24.60.252:38349 requested a full tablet report, sending...
I20260810 07:24:52.794745 321 ts_manager.cc:194] Registered new tserver with Master: 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999)
I20260810 07:24:52.800010 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.776941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:36775 (local address 127.24.60.253:35835)
0810 07:24:52.777013 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.777015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.777353 (+ 338us) server_negotiation.cc:407] Connection header received
0810 07:24:52.778128 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.778129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.778144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.778211 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.779040 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.779639 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781037 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.781366 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.781439 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.785453 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.785455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.785455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.785465 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.787842 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.788315 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.788317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.788317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.788389 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.788699 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.788700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.788700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.788838 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.788927 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.799733 (+ 10806us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.799897 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:24:52.801342 702 heartbeater.cc:347] Connected to a master server at 127.24.60.253:35835
I20260810 07:24:52.801426 702 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.801573 702 heartbeater.cc:511] Master 127.24.60.253:35835 requested a full tablet report, sending...
I20260810 07:24:52.802050 32559 ts_manager.cc:194] Registered new tserver with Master: 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999)
I20260810 07:24:52.804565 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:41733
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:45641
--webserver_port=44351
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:52.892423 708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:52.892669 708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:52.892697 708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:52.894009 708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:52.894068 708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:52.894097 708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:52.895745 708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:41733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=44351
--tserver_master_addrs=127.24.60.254:45641,127.24.60.253:35835,127.24.60.252:38349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:52.896108 708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:52.896387 708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:52.902349 714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.902370 716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:52.902514 713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:52.902985 708 server_base.cc:1034] running on GCE node
I20260810 07:24:52.903398 708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:52.903738 708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:52.905012 708 hybrid_clock.cc:648] HybridClock initialized: now 1786346692904998 us; error 34 us; skew 500 ppm
I20260810 07:24:52.905337 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.901772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.24.60.252:38349)
0810 07:24:52.901849 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.901852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.901884 (+ 32us) server_negotiation.cc:407] Connection header received
0810 07:24:52.901918 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.901919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.901949 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.902081 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:52.902261 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.903099 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.904448 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.904877 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.904966 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.905028 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.905221 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.905275 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:24:52.905391 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.901652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:33126)
0810 07:24:52.901772 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.901786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.901839 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.902088 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.902090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.902092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:52.902153 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.902155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.903214 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.903216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.904321 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.904326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.905006 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.905008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.905088 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.905093 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.905187 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.905252 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 07:24:52.905789 335 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 07:24:52.905982 371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac336ad4dd04ec5bbb549375c5f006b. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.906147 371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: This master's current role is: LEARNER
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:52.908246 708 init.cc:377] Logged in from keytab as oryx/127.24.60.195@KRBTEST.COM (short username oryx)
I20260810 07:24:52.908810 708 webserver.cc:460] Webserver started at http://127.24.60.195:44351/ using document root <none> and password file <none>
I20260810 07:24:52.909080 708 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:52.909194 708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:52.910754 708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:52.911545 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.911896 708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:52.912034 708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "48ef028d25ec4faeb78bd84a23eac893"
format_stamp: "Formatted at 2026-08-10 07:24:51 on dist-test-slave-vxj2"
server_key: "c58829613ff4ddd7ad2754c76534ccc8"
server_key_iv: "921752ceffda934d7f07ca33cc5877a4"
server_key_version: "encryptionkey@0"
I20260810 07:24:52.912458 708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:52.932641 708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:52.935324 708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:52.935470 708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:52.935685 708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:52.936244 708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:52.936285 708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.936311 708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:52.936381 708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:52.943125 708 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:41733
I20260810 07:24:52.943207 856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:41733 every 8 connection(s)
I20260810 07:24:52.943608 708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:52.950883 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 708
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.252@KRBTEST.COM
Aug 10 07:24:52 dist-test-slave-vxj2 krb5kdc[31808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346692, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.253@KRBTEST.COM
I20260810 07:24:52.956712 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.944534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:50631)
0810 07:24:52.944719 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.944767 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.945463 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.945735 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.945740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.946146 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.946379 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.946384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.947615 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.947616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.948605 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.948609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.949508 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.949596 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.950119 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.950124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.951967 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.954712 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.954725 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.954728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.955100 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.955359 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.955360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.955360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.955531 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.955856 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.955863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.956069 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.956189 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.956426 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260810 07:24:52.957094 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.944565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:50631 (local address 127.24.60.254:45641)
0810 07:24:52.944624 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.944627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.944894 (+ 267us) server_negotiation.cc:407] Connection header received
0810 07:24:52.945574 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.945575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.945590 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.945654 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.946482 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.947516 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949139 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.949443 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949511 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.952100 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.952103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.952104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.952112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.954606 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.955191 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.955192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.955193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.955266 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.955622 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.955623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.955624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.955760 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.955825 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.956889 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.956986 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 07:24:52.957665 858 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45641
I20260810 07:24:52.957794 858 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.958055 858 heartbeater.cc:511] Master 127.24.60.254:45641 requested a full tablet report, sending...
I20260810 07:24:52.958658 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.944555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:54563 (local address 127.24.60.252:38349)
0810 07:24:52.944615 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.944618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.944897 (+ 279us) server_negotiation.cc:407] Connection header received
0810 07:24:52.945656 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.945658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.945675 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.945747 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.946584 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.947540 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949470 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.949749 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949818 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.952952 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.952953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.952954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.952962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.957098 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.957714 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.957715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.957716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.957785 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.958006 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.958007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.958007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.958148 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.958218 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.958450 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.958555 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:24:52.958671 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.944534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.24.60.195:54563)
0810 07:24:52.944807 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.944816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.945480 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.946266 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.946267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.946433 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.946496 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.946498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.948714 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.948715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.949364 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.949367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.949829 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.949914 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.950267 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.950272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.952808 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.957419 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.957420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.957420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.957651 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.957863 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.957864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.957864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.957932 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.958247 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.958248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.958295 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.958386 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.958526 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260810 07:24:52.958711 31861 ts_manager.cc:194] Re-registered known tserver with Master: 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733)
I20260810 07:24:52.959270 31861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:50631
I20260810 07:24:52.959867 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.945063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:41245 (local address 127.24.60.253:35835)
0810 07:24:52.945136 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.945138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.945179 (+ 41us) server_negotiation.cc:407] Connection header received
0810 07:24:52.945719 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.945721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.945733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.945784 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.946526 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.947048 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.948700 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.949012 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949076 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.953876 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.953878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.953878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.953887 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.957867 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.958261 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.958262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.958263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.958600 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.959170 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.959171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.959171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.959296 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.959390 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.959653 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.959756 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:24:52.960208 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.944856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.24.60.195:41245)
0810 07:24:52.945006 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.945017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.945480 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.945834 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.945835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.946168 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:52.946380 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.946387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.947167 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.947168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.948605 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.948609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.949109 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.949110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.949289 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.949988 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:52.950000 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:52.953518 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:52.957953 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.957954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.957954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.958176 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.959016 (+ 840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:52.959017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:52.959017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:52.959075 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:52.959392 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:52.959394 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:52.959440 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.959534 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.960073 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260810 07:24:52.960253 857 heartbeater.cc:347] Connected to a master server at 127.24.60.252:38349
I20260810 07:24:52.960314 857 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.960897 859 heartbeater.cc:347] Connected to a master server at 127.24.60.253:35835
I20260810 07:24:52.960963 859 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:52.961022 859 heartbeater.cc:511] Master 127.24.60.253:35835 requested a full tablet report, sending...
I20260810 07:24:52.961220 857 heartbeater.cc:511] Master 127.24.60.252:38349 requested a full tablet report, sending...
I20260810 07:24:52.961431 32559 ts_manager.cc:194] Registered new tserver with Master: 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733)
I20260810 07:24:52.961549 321 ts_manager.cc:194] Registered new tserver with Master: 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733)
I20260810 07:24:52.963577 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.952015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.24.60.252:38349)
0810 07:24:52.952161 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.952162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.952342 (+ 180us) server_negotiation.cc:407] Connection header received
0810 07:24:52.952373 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.952374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.952391 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.952460 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.953473 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.954255 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.955347 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.955669 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.955727 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.957950 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.957951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.957952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.957960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.961955 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.962445 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.962448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.962448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.962521 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.962764 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.962766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.962766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.962914 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.963036 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.963351 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.963438 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260810 07:24:52.965955 870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.952685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.24.60.254:45641)
0810 07:24:52.953927 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.953929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.954086 (+ 157us) server_negotiation.cc:407] Connection header received
0810 07:24:52.954109 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.954110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.954133 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.954196 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.955087 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.955820 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.956912 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.957426 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.957567 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.962516 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.962519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.962519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.962529 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.964677 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.965044 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.965047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.965047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.965115 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.965384 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.965385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.965385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.965513 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.965620 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.965741 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.965836 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":83,"threads_started":1}
I20260810 07:24:52.967454 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.953623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.24.60.253:35835)
0810 07:24:52.956077 (+ 2454us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.956079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.960077 (+ 3998us) server_negotiation.cc:407] Connection header received
0810 07:24:52.960145 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.960146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.960159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.960230 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:52.960955 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.961712 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.962702 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.963038 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.963115 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.964326 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:52.964328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:52.964328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:52.964337 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:52.966270 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.966641 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.966643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.966643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.966715 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:52.966913 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:52.966914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:52.966915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:52.967040 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:52.967149 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.967250 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.967345 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2427,"thread_start_us":77,"threads_started":1}
I20260810 07:24:52.967749 31861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:52.969655 31861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:52.970566 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: NON_VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: true } } }
I20260810 07:24:52.970680 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.971834 877 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: attempting to promote NON_VOTER 1da4db7cdd3741d3b5468b919147bed1 to VOTER
I20260810 07:24:52.971992 877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:52.972419 32572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Refusing update from remote peer 0ac336ad4dd04ec5bbb549375c5f006b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 07:24:52.972628 876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 07:24:52.972683 335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 LEARNER]: Refusing update from remote peer 0ac336ad4dd04ec5bbb549375c5f006b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 07:24:52.972924 876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 07:24:52.973515 877 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.973667 32572 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.974020 335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.974617 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.974681 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.974730 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.974767 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.974881 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.974943 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:52.975183 371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.975296 371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:24:52.976366 886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:52.976476 886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:24:52.979661 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.975490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:40999 (local address 127.0.0.1:34954)
0810 07:24:52.975560 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.975576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.975630 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.976096 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.976096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.976098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:52.976153 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.976156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.977337 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.977338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.978452 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.978454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.979400 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.979401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.979467 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.979470 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.979528 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.979586 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 07:24:52.979677 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.975648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.24.60.194:40999)
0810 07:24:52.975894 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.975900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.975911 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:52.975962 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.975965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.975987 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.976100 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:52.976233 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.977255 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.978537 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.979318 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.979402 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.979480 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.979541 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.979583 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
I20260810 07:24:52.980558 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.975827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41733 (local address 127.0.0.1:44482)
0810 07:24:52.975949 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.975957 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.975996 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.976887 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.976888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.976890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:52.976958 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.976960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.978011 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.978013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.978892 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.978896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.980265 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.980266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.980350 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.980354 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.980439 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.980485 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":89,"threads_started":1}
I20260810 07:24:52.980746 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.975632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36675 (local address 127.0.0.1:55420)
0810 07:24:52.975799 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:52.975810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:52.975857 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:52.977088 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:52.977089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:52.977090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:52.977122 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.977123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.978419 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.978420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.979334 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:52.979337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.980533 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.980534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:52.980603 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.980607 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:52.980636 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 07:24:52.980676 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260810 07:24:52.980937 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.976141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.24.60.195:41733)
0810 07:24:52.976686 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.976690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.976700 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:52.976737 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.976740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.976760 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.976866 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:52.977042 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.977894 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.978996 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.979749 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.979823 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.980204 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.980461 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.980519 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":86,"threads_started":1}
I20260810 07:24:52.980934 625 tablet_service.cc:1467] Processing CreateTablet for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964 (DEFAULT_TABLE table=test-table [id=d7eefe9f95c04374907645ded455668a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:52.981480 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:52.975869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.24.60.193:36675)
0810 07:24:52.976400 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:52.976405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:52.976416 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:52.976998 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:52.977001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:52.977018 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:52.977092 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:52.977221 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.978312 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.979469 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:52.980422 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:52.980919 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:52.981302 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:52.981335 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 07:24:52.981398 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":98,"threads_started":1}
I20260810 07:24:52.981484 625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:52.981722 784 tablet_service.cc:1467] Processing CreateTablet for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964 (DEFAULT_TABLE table=test-table [id=d7eefe9f95c04374907645ded455668a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:52.982065 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:52.982059 454 tablet_service.cc:1467] Processing CreateTablet for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964 (DEFAULT_TABLE table=test-table [id=d7eefe9f95c04374907645ded455668a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:52.982409 454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:52.984987 893 tablet_bootstrap.cc:492] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Bootstrap starting.
I20260810 07:24:52.985846 893 tablet_bootstrap.cc:654] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:52.986234 893 log.cc:824] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:52.986398 895 tablet_bootstrap.cc:492] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Bootstrap starting.
I20260810 07:24:52.986771 894 tablet_bootstrap.cc:492] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Bootstrap starting.
I20260810 07:24:52.987135 893 tablet_bootstrap.cc:492] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: No bootstrap required, opened a new log
I20260810 07:24:52.987159 895 tablet_bootstrap.cc:654] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:52.987215 893 ts_tablet_manager.cc:1397] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:24:52.987483 895 log.cc:824] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:52.987511 894 tablet_bootstrap.cc:654] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:52.987900 894 log.cc:824] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:52.988306 895 tablet_bootstrap.cc:492] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: No bootstrap required, opened a new log
I20260810 07:24:52.988404 895 ts_tablet_manager.cc:1397] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:52.988852 894 tablet_bootstrap.cc:492] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: No bootstrap required, opened a new log
I20260810 07:24:52.988986 894 ts_tablet_manager.cc:1397] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:24:52.989320 893 raft_consensus.cc:348] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:52.989519 893 raft_consensus.cc:374] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:52.989565 893 raft_consensus.cc:729] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ef028d25ec4faeb78bd84a23eac893, State: Initialized, Role: FOLLOWER
I20260810 07:24:52.989691 893 consensus_queue.cc:260] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:52.989980 893 ts_tablet_manager.cc:1428] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:52.990097 858 heartbeater.cc:503] Master 127.24.60.254:45641 was elected leader, sending a full tablet report...
I20260810 07:24:52.990689 895 raft_consensus.cc:348] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:52.990875 895 raft_consensus.cc:374] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:52.990916 895 raft_consensus.cc:729] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6593755a904596852db7c05c2b91a4, State: Initialized, Role: FOLLOWER
I20260810 07:24:52.991074 895 consensus_queue.cc:260] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:52.991395 895 ts_tablet_manager.cc:1428] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:52.991566 540 heartbeater.cc:503] Master 127.24.60.254:45641 was elected leader, sending a full tablet report...
I20260810 07:24:52.993131 894 raft_consensus.cc:348] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:52.993368 894 raft_consensus.cc:374] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:52.993394 894 raft_consensus.cc:729] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da144e8a0bd4cc3aa5b46c40f8b6807, State: Initialized, Role: FOLLOWER
I20260810 07:24:52.993534 894 consensus_queue.cc:260] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:52.993885 894 ts_tablet_manager.cc:1428] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260810 07:24:52.993894 701 heartbeater.cc:503] Master 127.24.60.254:45641 was elected leader, sending a full tablet report...
W20260810 07:24:53.028141 703 tablet.cc:2369] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:53.088485 903 raft_consensus.cc:482] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:53.088655 903 raft_consensus.cc:504] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:53.089416 903 leader_election.cc:288] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999), 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733)
I20260810 07:24:53.092618 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.089645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:40999 (local address 127.24.60.193:48659)
0810 07:24:53.089890 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:53.089903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:53.089986 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:53.090165 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:53.090165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:53.090167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:53.090247 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:53.090249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.090914 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.090916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:53.091673 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:53.091675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.092333 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.092334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:53.092421 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.092430 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 07:24:53.092475 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 07:24:53.092513 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":198}
I20260810 07:24:53.092643 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.089735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:48659 (local address 127.24.60.194:40999)
0810 07:24:53.089867 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:53.089869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:53.089969 (+ 100us) server_negotiation.cc:407] Connection header received
0810 07:24:53.090063 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:53.090064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:53.090080 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:53.090159 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:53.090335 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.090817 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.091827 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.092229 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.092305 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.092315 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:53.092507 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 07:24:53.092566 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 07:24:53.092711 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.089749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:60075 (local address 127.24.60.195:41733)
0810 07:24:53.089888 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:53.089890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:53.089899 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:53.089931 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:53.089932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:53.089940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:53.090003 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:53.090189 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.090687 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.091809 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.092269 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.092364 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.092373 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:53.092585 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 07:24:53.092632 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:24:53.092795 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.089640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41733 (local address 127.24.60.193:60075)
0810 07:24:53.089707 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:53.089722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:53.089772 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:53.090023 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:53.090024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:53.090026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:53.090115 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:53.090117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.090780 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.090781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:53.091650 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:53.091654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.092393 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.092395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:53.092495 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.092499 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:53.092566 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:24:53.092610 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 07:24:53.093137 649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d45bd8ec9e4f4ab3b43cdc57daf964" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" is_pre_election: true
I20260810 07:24:53.093415 649 raft_consensus.cc:2454] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 0.
I20260810 07:24:53.093526 808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d45bd8ec9e4f4ab3b43cdc57daf964" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ef028d25ec4faeb78bd84a23eac893" is_pre_election: true
I20260810 07:24:53.093705 808 raft_consensus.cc:2454] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 0.
I20260810 07:24:53.093732 406 leader_election.cc:302] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6593755a904596852db7c05c2b91a4, 8da144e8a0bd4cc3aa5b46c40f8b6807; no voters:
I20260810 07:24:53.093914 903 raft_consensus.cc:2781] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:53.094002 903 raft_consensus.cc:482] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:53.094038 903 raft_consensus.cc:3037] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:53.094965 903 raft_consensus.cc:504] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:53.095144 903 leader_election.cc:288] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 election: Requested vote from peers 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999), 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733)
I20260810 07:24:53.095484 649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d45bd8ec9e4f4ab3b43cdc57daf964" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807"
I20260810 07:24:53.095578 649 raft_consensus.cc:3037] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:53.096434 649 raft_consensus.cc:2454] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 1.
I20260810 07:24:53.096597 808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d45bd8ec9e4f4ab3b43cdc57daf964" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ef028d25ec4faeb78bd84a23eac893"
I20260810 07:24:53.096666 406 leader_election.cc:302] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6593755a904596852db7c05c2b91a4, 8da144e8a0bd4cc3aa5b46c40f8b6807; no voters:
I20260810 07:24:53.096721 808 raft_consensus.cc:3037] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:53.096797 903 raft_consensus.cc:2781] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:53.096930 903 raft_consensus.cc:686] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 1 LEADER]: Becoming Leader. State: Replica: 6d6593755a904596852db7c05c2b91a4, State: Running, Role: LEADER
I20260810 07:24:53.097048 903 consensus_queue.cc:237] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } }
I20260810 07:24:53.097843 808 raft_consensus.cc:2454] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 1.
I20260810 07:24:53.098310 31861 catalog_manager.cc:5495] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d6593755a904596852db7c05c2b91a4 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "6d6593755a904596852db7c05c2b91a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } health_report { overall_health: UNKNOWN } } }
W20260810 07:24:53.107585 542 tablet.cc:2369] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:53.128666 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.125982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.24.60.193:36675)
0810 07:24:53.126051 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:53.126054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:53.126101 (+ 47us) server_negotiation.cc:407] Connection header received
0810 07:24:53.126150 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:53.126151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:53.126170 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:53.126247 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:53.126429 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.126930 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.127727 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.127937 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.128018 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.128247 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:53.128371 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:53.128476 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:53.128538 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 07:24:53.128563 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:24:53.131090 808 raft_consensus.cc:1261] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Refusing update from remote peer 6d6593755a904596852db7c05c2b91a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:53.131101 649 raft_consensus.cc:1261] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Refusing update from remote peer 6d6593755a904596852db7c05c2b91a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:53.131596 907 consensus_queue.cc:1035] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:53.131763 903 consensus_queue.cc:1035] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:53.135310 910 mvcc.cc:205] Tried to move back new op lower bound from 7316876055060951040 to 7316876054926663680. Current Snapshot: MvccSnapshot[applied={T|T < 7316876055060951040}]
I20260810 07:24:53.137419 911 mvcc.cc:205] Tried to move back new op lower bound from 7316876055060951040 to 7316876054926663680. Current Snapshot: MvccSnapshot[applied={T|T < 7316876055060951040 or (T in {7316876055060951040})}]
I20260810 07:24:53.139465 31860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:53.139600 31860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:53.141309 31860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ac336ad4dd04ec5bbb549375c5f006b: Sending DeleteTablet for 3 replicas of tablet c6d45bd8ec9e4f4ab3b43cdc57daf964
I20260810 07:24:53.141696 625 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:53 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:34954
I20260810 07:24:53.141984 454 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:53 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:55420
I20260810 07:24:53.142148 784 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d45bd8ec9e4f4ab3b43cdc57daf964 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:53 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:44482
I20260810 07:24:53.142292 922 tablet_replica.cc:330] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: stopping tablet replica
I20260810 07:24:53.142447 922 raft_consensus.cc:2229] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:53.142592 922 raft_consensus.cc:2258] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:53.142613 920 tablet_replica.cc:330] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: stopping tablet replica
I20260810 07:24:53.142817 923 tablet_replica.cc:330] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: stopping tablet replica
I20260810 07:24:53.142963 923 raft_consensus.cc:2229] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:53.143107 923 raft_consensus.cc:2258] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:53.143285 920 raft_consensus.cc:2229] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:53.143399 920 raft_consensus.cc:2258] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:53.143745 923 ts_tablet_manager.cc:1905] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:53.144189 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 31824
I20260810 07:24:53.146183 923 ts_tablet_manager.cc:1918] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:53.146410 923 log.cc:1196] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/c6d45bd8ec9e4f4ab3b43cdc57daf964
I20260810 07:24:53.146845 923 ts_tablet_manager.cc:1939] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 48ef028d25ec4faeb78bd84a23eac893: Deleting consensus metadata
I20260810 07:24:53.149227 922 ts_tablet_manager.cc:1905] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:53.149632 920 ts_tablet_manager.cc:1905] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:53.151631 920 ts_tablet_manager.cc:1918] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:53.151633 922 ts_tablet_manager.cc:1918] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:53.151731 922 log.cc:1196] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/c6d45bd8ec9e4f4ab3b43cdc57daf964
I20260810 07:24:53.151965 920 log.cc:1196] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/c6d45bd8ec9e4f4ab3b43cdc57daf964
I20260810 07:24:53.152076 922 ts_tablet_manager.cc:1939] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 6d6593755a904596852db7c05c2b91a4: Deleting consensus metadata
I20260810 07:24:53.152372 920 ts_tablet_manager.cc:1939] T c6d45bd8ec9e4f4ab3b43cdc57daf964 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Deleting consensus metadata
W20260810 07:24:53.155525 728 connection.cc:535] server connection from 127.0.0.1:44482 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 07:24:53.155527 407 connection.cc:535] server connection from 127.0.0.1:55420 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 07:24:53.155824 568 connection.cc:535] server connection from 127.0.0.1:34954 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:24:53.162004 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.143614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.24.60.252:38349)
0810 07:24:53.143667 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:53.143669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:53.143680 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:53.143707 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:53.143708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:53.143723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:53.143786 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:53.153842 (+ 10056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.154685 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.155891 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.156224 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.156274 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.158226 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:53.158228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:53.158228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:53.158237 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:53.160411 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:53.160988 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:53.160990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:53.160990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:53.161060 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:53.161324 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:53.161325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:53.161326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:53.161472 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:53.161554 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:53.161774 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 07:24:53.161898 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:24:53.167333 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.158103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.24.60.252:38349)
0810 07:24:53.158154 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:53.158157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:53.158339 (+ 182us) server_negotiation.cc:407] Connection header received
0810 07:24:53.158457 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:53.158457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:53.158473 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:53.158509 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:53.159450 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.160261 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.161482 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.161767 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.161810 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.163610 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:53.163612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:53.163612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:53.163622 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:53.165834 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:53.166307 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:53.166309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:53.166309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:53.166372 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:53.166627 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:53.166629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:53.166629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:53.166794 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:53.166888 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:53.167131 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 07:24:53.167234 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:24:53.167901 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:53.158318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.24.60.253:35835)
0810 07:24:53.158373 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:53.158375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:53.158516 (+ 141us) server_negotiation.cc:407] Connection header received
0810 07:24:53.158637 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:53.158639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:53.158656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:53.158688 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:53.159810 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.160545 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.161577 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:53.161891 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:53.161935 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:53.163887 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:53.163889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:53.163889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:53.163899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:53.166552 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:53.166953 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:53.166955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:53.166955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:53.167018 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:53.167234 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:53.167235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:53.167235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:53.167372 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:53.167453 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:53.167691 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 07:24:53.167813 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 07:24:54.136994 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.136679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:55129)
0810 07:24:54.136829 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.136933 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
W20260810 07:24:54.137250 858 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:45641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.138192 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.137923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:48527)
0810 07:24:54.138067 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.138153 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260810 07:24:54.138546 540 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:45641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.138881 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.138513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:36297)
0810 07:24:54.138745 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.138835 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":47,"threads_started":1}
W20260810 07:24:54.139160 701 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:45641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.642515 935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ac336ad4dd04ec5bbb549375c5f006b)
I20260810 07:24:54.642606 936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ac336ad4dd04ec5bbb549375c5f006b)
I20260810 07:24:54.642753 936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:54.642757 935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:54.643126 935 leader_election.cc:288] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641), 2cf16959c76b4f5b8e70446d3a07e900 (127.24.60.253:35835)
I20260810 07:24:54.643566 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.643294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59564)
0810 07:24:54.643475 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.643513 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260810 07:24:54.643657 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.643274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59562)
0810 07:24:54.643418 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.643480 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260810 07:24:54.643390 936 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641), 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252:38349)
W20260810 07:24:54.643808 32540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
W20260810 07:24:54.644129 302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.644542 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.644439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59580)
0810 07:24:54.644486 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.644512 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 07:24:54.644665 32540 leader_election.cc:334] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.644965 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.644810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59588)
0810 07:24:54.644868 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.644921 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 07:24:54.645113 302 leader_election.cc:334] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.647855 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.643656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.24.60.253:35835)
0810 07:24:54.643841 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:54.643844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:54.643853 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:54.643879 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:54.643879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:54.643903 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:54.643988 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:54.644234 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.645125 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.646784 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.647392 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.647499 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:54.647511 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:54.647715 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 07:24:54.647767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260810 07:24:54.647855 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.643452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.24.60.252:38349)
0810 07:24:54.643612 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:54.643615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:54.643958 (+ 343us) server_negotiation.cc:407] Connection header received
0810 07:24:54.643988 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:54.643989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:54.644001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:54.644066 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:54.644387 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.645127 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.646784 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.647392 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.647487 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:54.647501 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:54.647715 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 07:24:54.647767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260810 07:24:54.647861 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.643513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:35835 (local address 127.0.0.1:58536)
0810 07:24:54.643632 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.643643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:54.643710 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:54.644016 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:54.644018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:54.644022 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:54.644092 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:54.644094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.645260 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.645261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:54.646644 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:54.646648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.647517 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.647518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:54.647626 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:54.647631 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:54.647706 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:24:54.647757 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":49,"threads_started":1}
I20260810 07:24:54.647864 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.643295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:38349 (local address 127.0.0.1:39006)
0810 07:24:54.643661 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.643672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:54.643748 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:54.644118 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:54.644119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:54.644120 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:54.644193 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:54.644195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.645249 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.645250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:54.646644 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:54.646648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:54.647519 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:54.647520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:54.647626 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:54.647631 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:54.647706 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:24:54.647763 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":66,"threads_started":1}
I20260810 07:24:54.648303 32572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1da4db7cdd3741d3b5468b919147bed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2cf16959c76b4f5b8e70446d3a07e900" is_pre_election: true
I20260810 07:24:54.648437 32572 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da4db7cdd3741d3b5468b919147bed1 in term 1.
I20260810 07:24:54.648648 302 leader_election.cc:302] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da4db7cdd3741d3b5468b919147bed1, 2cf16959c76b4f5b8e70446d3a07e900; no voters: 0ac336ad4dd04ec5bbb549375c5f006b
I20260810 07:24:54.648634 335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf16959c76b4f5b8e70446d3a07e900" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1da4db7cdd3741d3b5468b919147bed1" is_pre_election: true
I20260810 07:24:54.648782 335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf16959c76b4f5b8e70446d3a07e900 in term 1.
I20260810 07:24:54.648914 935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 07:24:54.649008 935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0ac336ad4dd04ec5bbb549375c5f006b)
I20260810 07:24:54.649040 935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:24:54.649027 32539 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da4db7cdd3741d3b5468b919147bed1, 2cf16959c76b4f5b8e70446d3a07e900; no voters: 0ac336ad4dd04ec5bbb549375c5f006b
I20260810 07:24:54.649132 936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 07:24:54.649207 936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0ac336ad4dd04ec5bbb549375c5f006b)
I20260810 07:24:54.649227 936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:24:54.649919 936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:54.649981 935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:54.650075 936 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 2 election: Requested vote from peers 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641), 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252:38349)
I20260810 07:24:54.650166 935 leader_election.cc:288] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [CANDIDATE]: Term 2 election: Requested vote from peers 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641), 2cf16959c76b4f5b8e70446d3a07e900 (127.24.60.253:35835)
I20260810 07:24:54.650337 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.650174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59594)
0810 07:24:54.650260 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.650303 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 07:24:54.650334 335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf16959c76b4f5b8e70446d3a07e900" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1da4db7cdd3741d3b5468b919147bed1"
I20260810 07:24:54.650377 32572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1da4db7cdd3741d3b5468b919147bed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2cf16959c76b4f5b8e70446d3a07e900"
I20260810 07:24:54.650468 32572 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1da4db7cdd3741d3b5468b919147bed1 in current term 2: Already voted for candidate 2cf16959c76b4f5b8e70446d3a07e900 in this term.
I20260810 07:24:54.650470 335 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2cf16959c76b4f5b8e70446d3a07e900 in current term 2: Already voted for candidate 1da4db7cdd3741d3b5468b919147bed1 in this term.
I20260810 07:24:54.650484 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.650241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59610)
0810 07:24:54.650318 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.650346 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 07:24:54.650974 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.650851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59620)
0810 07:24:54.650886 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.650945 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 07:24:54.651000 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:54.650887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59636)
0810 07:24:54.650939 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:54.650968 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 07:24:54.651057 302 leader_election.cc:334] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
W20260810 07:24:54.651072 32540 leader_election.cc:334] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:54.651094 302 leader_election.cc:302] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1da4db7cdd3741d3b5468b919147bed1; no voters: 0ac336ad4dd04ec5bbb549375c5f006b, 2cf16959c76b4f5b8e70446d3a07e900
I20260810 07:24:54.651115 32540 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cf16959c76b4f5b8e70446d3a07e900; no voters: 0ac336ad4dd04ec5bbb549375c5f006b, 1da4db7cdd3741d3b5468b919147bed1
I20260810 07:24:54.651178 935 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 07:24:54.651213 936 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 07:24:55.138027 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:55.137700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:50797)
0810 07:24:55.137886 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:55.137948 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260810 07:24:55.139213 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:55.138935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:44891)
0810 07:24:55.139082 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:55.139137 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260810 07:24:55.139765 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:55.139546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:42279)
0810 07:24:55.139677 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:55.139720 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260810 07:24:56.138861 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.138573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:58355)
0810 07:24:56.138739 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.138791 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260810 07:24:56.140075 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.139819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:33135)
0810 07:24:56.139979 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.140023 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
I20260810 07:24:56.140874 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.140357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:45183)
0810 07:24:56.140680 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.140760 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
I20260810 07:24:56.365492 954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:56.365624 954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:56.365876 954 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641), 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252:38349)
I20260810 07:24:56.366214 335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf16959c76b4f5b8e70446d3a07e900" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1da4db7cdd3741d3b5468b919147bed1" is_pre_election: true
I20260810 07:24:56.366300 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.366070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59934)
0810 07:24:56.366210 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.366254 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260810 07:24:56.366346 335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf16959c76b4f5b8e70446d3a07e900 in term 2.
I20260810 07:24:56.366652 32539 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da4db7cdd3741d3b5468b919147bed1, 2cf16959c76b4f5b8e70446d3a07e900; no voters:
I20260810 07:24:56.366796 954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 07:24:56.366870 954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:56.366883 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.366766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59946)
0810 07:24:56.366813 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.366849 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 07:24:56.366888 954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 2 FOLLOWER]: Advancing to term 3
W20260810 07:24:56.367046 32540 leader_election.cc:334] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:56.367862 954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:56.368139 954 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 3 election: Requested vote from peers 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641), 1da4db7cdd3741d3b5468b919147bed1 (127.24.60.252:38349)
I20260810 07:24:56.368304 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.368140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59962)
0810 07:24:56.368207 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.368250 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:24:56.368312 335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf16959c76b4f5b8e70446d3a07e900" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1da4db7cdd3741d3b5468b919147bed1"
I20260810 07:24:56.368433 335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:24:56.368752 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.368611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59974)
0810 07:24:56.368701 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.368731 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 07:24:56.368834 32540 leader_election.cc:334] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
I20260810 07:24:56.369484 335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf16959c76b4f5b8e70446d3a07e900 in term 3.
I20260810 07:24:56.369706 32539 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da4db7cdd3741d3b5468b919147bed1, 2cf16959c76b4f5b8e70446d3a07e900; no voters: 0ac336ad4dd04ec5bbb549375c5f006b
I20260810 07:24:56.369994 954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 07:24:56.370168 954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [term 3 LEADER]: Becoming Leader. State: Replica: 2cf16959c76b4f5b8e70446d3a07e900, State: Running, Role: LEADER
I20260810 07:24:56.370260 954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } }
I20260810 07:24:56.370671 957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf16959c76b4f5b8e70446d3a07e900. Latest consensus state: current_term: 3 leader_uuid: "2cf16959c76b4f5b8e70446d3a07e900" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:56.370757 957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:56.370961 959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:56.371335 959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:56.371392 959 catalog_manager.cc:1259] Loaded cluster ID: 3666b311175d451e981f62823ebc0070
I20260810 07:24:56.371407 959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:56.371560 959 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:56.371619 959 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Loaded TSK: 0
I20260810 07:24:56.371711 959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:24:56.441903 32558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:56.445451 32558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:56.447131 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.446959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:59992)
0810 07:24:56.447032 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.447082 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 07:24:56.447234 335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [term 3 FOLLOWER]: Refusing update from remote peer 2cf16959c76b4f5b8e70446d3a07e900: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 07:24:56.447640 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.447395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.0.0.1:60006)
0810 07:24:56.447571 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.447604 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 07:24:56.447567 957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 07:24:56.447849 32540 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 -> Peer 0ac336ad4dd04ec5bbb549375c5f006b (127.24.60.254:45641): Couldn't send request to peer 0ac336ad4dd04ec5bbb549375c5f006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 07:24:56.448560 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf16959c76b4f5b8e70446d3a07e900. Latest consensus state: current_term: 3 leader_uuid: "2cf16959c76b4f5b8e70446d3a07e900" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:56.448705 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:24:56.449361 957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2cf16959c76b4f5b8e70446d3a07e900" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:56.449450 960 mvcc.cc:205] Tried to move back new op lower bound from 7316876068644048896 to 7316876068333449216. Current Snapshot: MvccSnapshot[applied={T|T < 7316876068644048896}]
I20260810 07:24:56.449498 957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:56.449533 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2cf16959c76b4f5b8e70446d3a07e900" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0ac336ad4dd04ec5bbb549375c5f006b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45641 } } peers { permanent_uuid: "2cf16959c76b4f5b8e70446d3a07e900" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 35835 } attrs { promote: false } } peers { permanent_uuid: "1da4db7cdd3741d3b5468b919147bed1" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 38349 } attrs { promote: false } } }
I20260810 07:24:56.449643 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da4db7cdd3741d3b5468b919147bed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:24:56.456565 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.452675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.24.60.194:40999)
0810 07:24:56.452808 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:56.452811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:56.452820 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:56.452847 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:56.452848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:56.452859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:56.452941 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:56.453252 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.454057 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.455565 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.456139 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.456233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.456248 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:56.456424 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 07:24:56.456471 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260810 07:24:56.456565 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.452449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:40999 (local address 127.0.0.1:35194)
0810 07:24:56.452689 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.452701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:56.452747 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:56.453005 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:56.453006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:56.453008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:56.453120 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:56.453121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.454161 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.454162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:56.455475 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:56.455479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.456234 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.456235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:56.456322 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.456328 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:24:56.456416 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 07:24:56.456467 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":37,"thread_start_us":114,"threads_started":1}
I20260810 07:24:56.456831 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.452412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41733 (local address 127.0.0.1:39734)
0810 07:24:56.452462 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.452475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:56.452538 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:56.453026 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:56.453026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:56.453027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:56.453119 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:56.453121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.454297 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.454298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:56.455582 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:56.455585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.456562 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.456563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:56.456667 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.456671 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:56.456704 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 07:24:56.456751 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 07:24:56.457223 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.452691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.24.60.195:41733)
0810 07:24:56.452819 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:56.452822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:56.452833 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:56.452866 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:56.452867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:56.452880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:56.453024 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:56.453238 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.454184 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.455702 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.456460 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.456929 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.456944 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:56.456969 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 07:24:56.457017 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260810 07:24:56.457607 625 tablet_service.cc:1467] Processing CreateTablet for tablet 83959512cee34d25b34a2acade6adf51 (DEFAULT_TABLE table=test-table [id=a8cf6c40301b4948a28f1609427dae5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:56.457597 784 tablet_service.cc:1467] Processing CreateTablet for tablet 83959512cee34d25b34a2acade6adf51 (DEFAULT_TABLE table=test-table [id=a8cf6c40301b4948a28f1609427dae5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:56.457758 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83959512cee34d25b34a2acade6adf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:56.457758 625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83959512cee34d25b34a2acade6adf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:56.458119 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.452533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36675 (local address 127.0.0.1:43314)
0810 07:24:56.453040 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.453051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:56.453116 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:56.453448 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:56.453449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:56.453451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:56.453502 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:56.453503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.454550 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.454551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:56.456184 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:56.456187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.457801 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.457802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:56.457895 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.457901 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:24:56.457958 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 07:24:56.458013 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":83,"threads_started":1}
I20260810 07:24:56.458503 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.452813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.24.60.193:36675)
0810 07:24:56.452973 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:56.452976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:56.453232 (+ 256us) server_negotiation.cc:407] Connection header received
0810 07:24:56.453268 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:56.453270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:56.453284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:56.453353 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:56.453594 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.454455 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.456304 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.457667 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.457746 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.457765 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:56.458264 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 07:24:56.458342 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260810 07:24:56.459501 973 tablet_bootstrap.cc:492] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Bootstrap starting.
I20260810 07:24:56.459561 972 tablet_bootstrap.cc:492] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Bootstrap starting.
I20260810 07:24:56.459918 454 tablet_service.cc:1467] Processing CreateTablet for tablet 83959512cee34d25b34a2acade6adf51 (DEFAULT_TABLE table=test-table [id=a8cf6c40301b4948a28f1609427dae5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:56.460052 454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83959512cee34d25b34a2acade6adf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:56.460088 972 tablet_bootstrap.cc:654] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:56.460088 973 tablet_bootstrap.cc:654] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:56.460795 972 tablet_bootstrap.cc:492] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: No bootstrap required, opened a new log
I20260810 07:24:56.460855 972 ts_tablet_manager.cc:1397] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:56.461009 972 raft_consensus.cc:348] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.461102 972 raft_consensus.cc:374] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:56.461128 972 raft_consensus.cc:729] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da144e8a0bd4cc3aa5b46c40f8b6807, State: Initialized, Role: FOLLOWER
I20260810 07:24:56.461231 972 consensus_queue.cc:260] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.461488 972 ts_tablet_manager.cc:1428] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:56.461910 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.461752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:49353)
0810 07:24:56.461841 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.461872 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 07:24:56.462553 973 tablet_bootstrap.cc:492] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: No bootstrap required, opened a new log
I20260810 07:24:56.462674 973 ts_tablet_manager.cc:1397] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:24:56.462925 977 tablet_bootstrap.cc:492] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4: Bootstrap starting.
I20260810 07:24:56.462908 973 raft_consensus.cc:348] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.463100 973 raft_consensus.cc:374] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:56.463152 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.463032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:44139)
0810 07:24:56.463081 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.463110 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 07:24:56.463169 973 raft_consensus.cc:729] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ef028d25ec4faeb78bd84a23eac893, State: Initialized, Role: FOLLOWER
I20260810 07:24:56.463315 702 heartbeater.cc:503] Master 127.24.60.253:35835 was elected leader, sending a full tablet report...
I20260810 07:24:56.463326 977 tablet_bootstrap.cc:654] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:56.463330 973 consensus_queue.cc:260] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.463707 973 ts_tablet_manager.cc:1428] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:56.464023 977 tablet_bootstrap.cc:492] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4: No bootstrap required, opened a new log
I20260810 07:24:56.464097 977 ts_tablet_manager.cc:1397] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:56.464248 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.464088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:47175)
0810 07:24:56.464182 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.464214 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 07:24:56.464262 977 raft_consensus.cc:348] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.464351 977 raft_consensus.cc:374] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:56.464385 977 raft_consensus.cc:729] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6593755a904596852db7c05c2b91a4, State: Initialized, Role: FOLLOWER
I20260810 07:24:56.464438 977 consensus_queue.cc:260] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.464620 977 ts_tablet_manager.cc:1428] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:56.465094 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.464954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:42311)
0810 07:24:56.465021 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.465060 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 07:24:56.465764 859 heartbeater.cc:503] Master 127.24.60.253:35835 was elected leader, sending a full tablet report...
I20260810 07:24:56.466429 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.466209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:53431)
0810 07:24:56.466341 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.466372 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 07:24:56.498895 981 raft_consensus.cc:482] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:56.499017 981 raft_consensus.cc:504] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.499266 981 leader_election.cc:288] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733), 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999)
I20260810 07:24:56.499568 808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83959512cee34d25b34a2acade6adf51" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ef028d25ec4faeb78bd84a23eac893" is_pre_election: true
I20260810 07:24:56.499598 649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83959512cee34d25b34a2acade6adf51" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" is_pre_election: true
I20260810 07:24:56.499678 808 raft_consensus.cc:2454] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 0.
I20260810 07:24:56.499707 649 raft_consensus.cc:2454] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 0.
I20260810 07:24:56.499879 405 leader_election.cc:302] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ef028d25ec4faeb78bd84a23eac893, 6d6593755a904596852db7c05c2b91a4; no voters:
I20260810 07:24:56.500043 981 raft_consensus.cc:2781] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:56.500088 981 raft_consensus.cc:482] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:56.500118 981 raft_consensus.cc:3037] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:56.500931 981 raft_consensus.cc:504] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.501267 981 leader_election.cc:288] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 election: Requested vote from peers 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733), 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999)
I20260810 07:24:56.501498 649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83959512cee34d25b34a2acade6adf51" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807"
I20260810 07:24:56.501621 649 raft_consensus.cc:3037] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:56.501562 808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83959512cee34d25b34a2acade6adf51" candidate_uuid: "6d6593755a904596852db7c05c2b91a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ef028d25ec4faeb78bd84a23eac893"
I20260810 07:24:56.501682 808 raft_consensus.cc:3037] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:56.502393 649 raft_consensus.cc:2454] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 1.
I20260810 07:24:56.502475 808 raft_consensus.cc:2454] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6593755a904596852db7c05c2b91a4 in term 1.
I20260810 07:24:56.502625 406 leader_election.cc:302] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6593755a904596852db7c05c2b91a4, 8da144e8a0bd4cc3aa5b46c40f8b6807; no voters:
I20260810 07:24:56.502754 981 raft_consensus.cc:2781] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:56.502914 981 raft_consensus.cc:686] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 1 LEADER]: Becoming Leader. State: Replica: 6d6593755a904596852db7c05c2b91a4, State: Running, Role: LEADER
I20260810 07:24:56.502966 981 consensus_queue.cc:237] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } }
I20260810 07:24:56.503155 541 heartbeater.cc:503] Master 127.24.60.253:35835 was elected leader, sending a full tablet report...
I20260810 07:24:56.503314 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.503189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:54207)
0810 07:24:56.503243 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.503276 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 07:24:56.503893 32558 catalog_manager.cc:5495] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d6593755a904596852db7c05c2b91a4 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "6d6593755a904596852db7c05c2b91a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d6593755a904596852db7c05c2b91a4" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:56.520874 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.518094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.24.60.193:36675)
0810 07:24:56.518159 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:56.518160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:56.518170 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:56.518198 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:56.518199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:56.518208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:56.518256 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:56.518438 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.518923 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.520065 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.520282 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.520357 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.520563 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:56.520599 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:56.520654 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:56.520732 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 07:24:56.520759 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:24:56.522154 649 raft_consensus.cc:1261] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Refusing update from remote peer 6d6593755a904596852db7c05c2b91a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:56.522156 808 raft_consensus.cc:1261] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Refusing update from remote peer 6d6593755a904596852db7c05c2b91a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:56.522423 981 consensus_queue.cc:1035] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ef028d25ec4faeb78bd84a23eac893" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:56.522544 982 consensus_queue.cc:1035] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da144e8a0bd4cc3aa5b46c40f8b6807" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:56.523663 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.523510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:39619)
0810 07:24:56.523588 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.523621 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 07:24:56.524353 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.524209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:58953)
0810 07:24:56.524268 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.524313 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 07:24:56.525759 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.525470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.194:51207)
0810 07:24:56.525678 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.525715 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260810 07:24:56.526790 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.526657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:44655)
0810 07:24:56.526712 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.526753 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 07:24:56.527029 32558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:56.527091 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.526987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.193:50147)
0810 07:24:56.527031 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.527065 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 07:24:56.527128 32558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:56.527745 989 mvcc.cc:205] Tried to move back new op lower bound from 7316876068952674304 to 7316876068876935168. Current Snapshot: MvccSnapshot[applied={T|T < 7316876068952674304 or (T in {7316876068952674304})}]
I20260810 07:24:56.528590 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.528461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45641 (local address 127.24.60.195:58021)
0810 07:24:56.528513 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.528555 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:45641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 07:24:56.529137 32558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cf16959c76b4f5b8e70446d3a07e900: Sending DeleteTablet for 3 replicas of tablet 83959512cee34d25b34a2acade6adf51
I20260810 07:24:56.529564 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 83959512cee34d25b34a2acade6adf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:56 UTC) from {username='oryx', principal='oryx/127.24.60.253@KRBTEST.COM'} at 127.0.0.1:39734
I20260810 07:24:56.529623 625 tablet_service.cc:1514] Processing DeleteTablet for tablet 83959512cee34d25b34a2acade6adf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:56 UTC) from {username='oryx', principal='oryx/127.24.60.253@KRBTEST.COM'} at 127.0.0.1:35194
I20260810 07:24:56.529887 997 tablet_replica.cc:330] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: stopping tablet replica
I20260810 07:24:56.529964 997 raft_consensus.cc:2229] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:56.530035 997 raft_consensus.cc:2258] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:56.530100 454 tablet_service.cc:1514] Processing DeleteTablet for tablet 83959512cee34d25b34a2acade6adf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:56 UTC) from {username='oryx', principal='oryx/127.24.60.253@KRBTEST.COM'} at 127.0.0.1:43314
I20260810 07:24:56.530234 998 tablet_replica.cc:330] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: stopping tablet replica
I20260810 07:24:56.530318 999 tablet_replica.cc:330] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4: stopping tablet replica
I20260810 07:24:56.530395 999 raft_consensus.cc:2229] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:56.530352 998 raft_consensus.cc:2229] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:56.530470 999 raft_consensus.cc:2258] T 83959512cee34d25b34a2acade6adf51 P 6d6593755a904596852db7c05c2b91a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:56.530483 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 32740
I20260810 07:24:56.530616 997 ts_tablet_manager.cc:1905] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:56.534592 997 ts_tablet_manager.cc:1918] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:56.534682 997 log.cc:1196] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/83959512cee34d25b34a2acade6adf51
I20260810 07:24:56.535168 997 ts_tablet_manager.cc:1939] T 83959512cee34d25b34a2acade6adf51 P 8da144e8a0bd4cc3aa5b46c40f8b6807: Deleting consensus metadata
I20260810 07:24:56.536379 32540 catalog_manager.cc:4841] TS 8da144e8a0bd4cc3aa5b46c40f8b6807 (127.24.60.194:40999): tablet 83959512cee34d25b34a2acade6adf51 (table test-table [id=a8cf6c40301b4948a28f1609427dae5a]) successfully deleted
I20260810 07:24:56.536823 998 raft_consensus.cc:2258] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:56.540123 998 ts_tablet_manager.cc:1905] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:56.542620 998 ts_tablet_manager.cc:1918] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:56.542824 998 log.cc:1196] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/83959512cee34d25b34a2acade6adf51
I20260810 07:24:56.543180 998 ts_tablet_manager.cc:1939] T 83959512cee34d25b34a2acade6adf51 P 48ef028d25ec4faeb78bd84a23eac893: Deleting consensus metadata
I20260810 07:24:56.543730 32539 catalog_manager.cc:4841] TS 48ef028d25ec4faeb78bd84a23eac893 (127.24.60.195:41733): tablet 83959512cee34d25b34a2acade6adf51 (table test-table [id=a8cf6c40301b4948a28f1609427dae5a]) successfully deleted
I20260810 07:24:56.544291 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.544175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36675 (local address 127.0.0.1:43328)
0810 07:24:56.544222 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.544253 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260810 07:24:56.544485 32539 catalog_manager.cc:4584] TS 6d6593755a904596852db7c05c2b91a4 (127.24.60.193:36675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83959512cee34d25b34a2acade6adf51: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36675: connect: Connection refused (error 111)
I20260810 07:24:56.545001 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 547
I20260810 07:24:56.553381 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 708
I20260810 07:24:56.566206 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 32405
I20260810 07:24:56.574257 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 32622
2026-08-10T07:24:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6736 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:56 dist-test-slave-vxj2 krb5kdc[1002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:56 dist-test-slave-vxj2 krb5kdc[1002](info): set up 2 sockets
Aug 10 07:24:56 dist-test-slave-vxj2 krb5kdc[1002](info): commencing operation
krb5kdc: starting...
W20260810 07:24:57.721513 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:24:57 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab.
Entry for principal oryx/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab.
I20260810 07:24:57.857353 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45683
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab
--principal=oryx/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:57.944417 1018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:57.944661 1018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:57.944687 1018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:57.945966 1018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:57.946015 1018 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 07:24:57.946029 1018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:57.946043 1018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:57.946055 1018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:57.946069 1018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:57.947711 1018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:45683
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.254.keytab
--principal=oryx/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:57.948062 1018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:57.948316 1018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:57.953539 1025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:57.953579 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:57.953522 1024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:57.953792 1018 server_base.cc:1034] running on GCE node
I20260810 07:24:57.954209 1018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:57.954516 1018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:57.955682 1018 hybrid_clock.cc:648] HybridClock initialized: now 1786346697955660 us; error 41 us; skew 500 ppm
Aug 10 07:24:57 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:57.958353 1018 init.cc:377] Logged in from keytab as oryx/127.24.60.254@KRBTEST.COM (short username oryx)
I20260810 07:24:57.958797 1018 webserver.cc:460] Webserver started at http://127.24.60.254:37309/ using document root <none> and password file <none>
I20260810 07:24:57.959060 1018 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:57.959108 1018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:57.959206 1018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:57.960304 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "8cc73832166f4f6288094fd3ad6f5a97"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-vxj2"
server_key: "ed6cddbe69e431c42724f51d2b6071e0"
server_key_iv: "6203666b43b2dcf6f4957eaec871824a"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.960623 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "8cc73832166f4f6288094fd3ad6f5a97"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-vxj2"
server_key: "ed6cddbe69e431c42724f51d2b6071e0"
server_key_iv: "6203666b43b2dcf6f4957eaec871824a"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.962139 1018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:57.963089 1037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.963295 1018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:57.963389 1018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "8cc73832166f4f6288094fd3ad6f5a97"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-vxj2"
server_key: "ed6cddbe69e431c42724f51d2b6071e0"
server_key_iv: "6203666b43b2dcf6f4957eaec871824a"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.963455 1018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:57.974017 1018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:57.976564 1018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:57.976699 1018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:57.981920 1018 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:45683
I20260810 07:24:57.982112 1108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:45683 every 8 connection(s)
I20260810 07:24:57.982342 1018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:57.983124 1109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:57.983189 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 1018
I20260810 07:24:57.983325 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:24:57.983577 24819 external_mini_cluster.cc:1428] Setting key c746f79443ce1bee0d0edf37014a5bca
I20260810 07:24:57.986752 1109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Bootstrap starting.
I20260810 07:24:57.987564 1109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:57.987905 1109 log.cc:824] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:57.988735 1109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: No bootstrap required, opened a new log
Aug 10 07:24:57 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:57.991312 1109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc73832166f4f6288094fd3ad6f5a97" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45683 } }
I20260810 07:24:57.991508 1109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:57.991559 1109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc73832166f4f6288094fd3ad6f5a97, State: Initialized, Role: FOLLOWER
I20260810 07:24:57.991703 1109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc73832166f4f6288094fd3ad6f5a97" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45683 } }
I20260810 07:24:57.991806 1109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:57.991852 1109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:57.991911 1109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:57.992512 1109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc73832166f4f6288094fd3ad6f5a97" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45683 } }
I20260810 07:24:57.992640 1109 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc73832166f4f6288094fd3ad6f5a97; no voters:
I20260810 07:24:57.992879 1109 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:57.992905 1116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:57.993023 1116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc73832166f4f6288094fd3ad6f5a97, State: Running, Role: LEADER
I20260810 07:24:57.993202 1109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:57.993124 1116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc73832166f4f6288094fd3ad6f5a97" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45683 } }
I20260810 07:24:57.993569 1118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc73832166f4f6288094fd3ad6f5a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc73832166f4f6288094fd3ad6f5a97" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45683 } } }
I20260810 07:24:57.993847 1118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:57.993760 1120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc73832166f4f6288094fd3ad6f5a97. Latest consensus state: current_term: 1 leader_uuid: "8cc73832166f4f6288094fd3ad6f5a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc73832166f4f6288094fd3ad6f5a97" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45683 } } }
I20260810 07:24:57.994167 1120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:57.994391 1127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:57.995038 1127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:57.996100 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.984187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.24.60.254:45683)
0810 07:24:57.984357 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:57.984366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:57.984393 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:24:57.984837 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:57.984845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:57.984896 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:57.985084 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:57.986002 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.987018 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.988097 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.988511 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.988604 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.991421 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:57.991426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:57.991426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:57.991442 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:57.993589 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.994087 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.994091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.994091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.994205 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.994458 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.994459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.994460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.994829 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:57.995005 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:57.995707 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 07:24:57.995897 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260810 07:24:57.997254 1127 catalog_manager.cc:1347] Generated new cluster ID: 3c5947eceed54ee5ab1fa5508bf8d30c
I20260810 07:24:57.997319 1127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:58.011492 1127 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:58.012123 1127 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:58.016795 1127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Generated new TSK 0
I20260810 07:24:58.017033 1127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab.
Entry for principal oryx/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab.
I20260810 07:24:58.052701 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45683
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:58.140035 1139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:58.140275 1139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:58.140303 1139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:58.141650 1139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:58.141703 1139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:58.141726 1139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:58.141744 1139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:24:58.143426 1139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:58.143703 1139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:58.143962 1139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:58.149268 1147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:58.149268 1149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:58.149293 1146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:58.149737 1139 server_base.cc:1034] running on GCE node
I20260810 07:24:58.149962 1139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:58.150202 1139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:58.151381 1139 hybrid_clock.cc:648] HybridClock initialized: now 1786346698151369 us; error 38 us; skew 500 ppm
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:58.154225 1139 init.cc:377] Logged in from keytab as oryx/127.24.60.193@KRBTEST.COM (short username oryx)
I20260810 07:24:58.154758 1139 webserver.cc:460] Webserver started at http://127.24.60.193:43749/ using document root <none> and password file <none>
I20260810 07:24:58.155016 1139 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:58.155086 1139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:58.155233 1139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:58.156337 1139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "29a8cd4ab84b49e49daa7ec296c646d7"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "8388bec8cb2e35d810e3321c2185d8ec"
server_key_iv: "8056c650eaccdc051211b08b2137e154"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.156724 1139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "29a8cd4ab84b49e49daa7ec296c646d7"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "8388bec8cb2e35d810e3321c2185d8ec"
server_key_iv: "8056c650eaccdc051211b08b2137e154"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.158190 1139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:24:58.159060 1157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.159245 1139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:58.159349 1139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "29a8cd4ab84b49e49daa7ec296c646d7"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "8388bec8cb2e35d810e3321c2185d8ec"
server_key_iv: "8056c650eaccdc051211b08b2137e154"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.159457 1139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:58.178068 1139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:58.180548 1139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:58.180732 1139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:58.181464 1139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:58.181527 1139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.181564 1139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:58.181595 1139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:58.190398 1139 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:45077
I20260810 07:24:58.190547 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:45077 every 8 connection(s)
I20260810 07:24:58.190876 1139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:58.193372 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.181876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:43737 (local address 127.24.60.254:45683)
0810 07:24:58.181945 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.181948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.182298 (+ 350us) server_negotiation.cc:407] Connection header received
0810 07:24:58.183089 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.183090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.183101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.183194 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.184224 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.185006 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.186943 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.187289 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.187363 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.190361 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.190363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.190363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.190373 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.191967 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.192393 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.192394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.192394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.192473 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.192698 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.192699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.192699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.192813 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.192864 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.193213 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.193302 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:24:58.193579 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.181884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.24.60.193:43737)
0810 07:24:58.182129 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.182184 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.182964 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.183294 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.183300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.183743 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:58.184109 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.184116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.185110 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.185111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.186851 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.186854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.187415 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.187416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.187584 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.188265 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:58.188276 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:58.190214 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:58.192059 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.192067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.192069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.192320 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.192537 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.192537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.192538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.192643 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.192868 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:58.192873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:58.192998 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.193208 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.193363 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":134,"threads_started":1}
I20260810 07:24:58.194198 1291 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45683
I20260810 07:24:58.194308 1291 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:58.194499 1291 heartbeater.cc:511] Master 127.24.60.254:45683 requested a full tablet report, sending...
I20260810 07:24:58.195154 1059 ts_manager.cc:194] Registered new tserver with Master: 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077)
I20260810 07:24:58.195760 1059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:43737
I20260810 07:24:58.199146 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 1139
I20260810 07:24:58.199256 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:58.199477 24819 external_mini_cluster.cc:1428] Setting key a9a294e2e1041ff23ac918360baff2c6
I20260810 07:24:58.202906 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.196614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58437 (local address 127.24.60.254:45683)
0810 07:24:58.196657 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.196659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.196795 (+ 136us) server_negotiation.cc:407] Connection header received
0810 07:24:58.196824 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.196825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.196835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.196883 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.197245 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.197725 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.198409 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.198594 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.198654 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.199829 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.199830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.199831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.199839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.201606 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.202076 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.202077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.202078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.202143 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.202336 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.202337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.202337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.202465 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.202540 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.202705 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.202803 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 07:24:58.202922 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.196545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.24.60.193:58437)
0810 07:24:58.196724 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.196734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.196787 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.196885 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.196885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.197071 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:58.197134 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.197136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.197812 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.197813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.198332 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.198334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.198669 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.198669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.198718 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.199305 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:58.199311 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:58.199750 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:58.201707 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.201709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.201709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.201960 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.202215 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.202215 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.202216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.202265 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.202543 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:58.202544 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:58.202596 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.202679 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.202800 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab.
Entry for principal oryx/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab.
I20260810 07:24:58.229645 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45683
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:58.318011 1300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:58.318255 1300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:58.318284 1300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:58.319483 1300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:58.319525 1300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:58.319547 1300 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:58.319566 1300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:24:58.321235 1300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:58.321462 1300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:58.321662 1300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:58.327275 1306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:58.327291 1309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:58.327330 1300 server_base.cc:1034] running on GCE node
W20260810 07:24:58.327291 1307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:58.327710 1300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:58.327966 1300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:58.329144 1300 hybrid_clock.cc:648] HybridClock initialized: now 1786346698329130 us; error 31 us; skew 500 ppm
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:58.332136 1300 init.cc:377] Logged in from keytab as oryx/127.24.60.194@KRBTEST.COM (short username oryx)
I20260810 07:24:58.332685 1300 webserver.cc:460] Webserver started at http://127.24.60.194:38509/ using document root <none> and password file <none>
I20260810 07:24:58.332958 1300 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:58.333007 1300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:58.333146 1300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:58.334367 1300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "8b0541e5ccc74b63907465503853adb3"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "6c37e9841fe3d011c6144ccd5a929b8b"
server_key_iv: "929e2dbb3c60bf3faa777c4a221649eb"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.334743 1300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "8b0541e5ccc74b63907465503853adb3"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "6c37e9841fe3d011c6144ccd5a929b8b"
server_key_iv: "929e2dbb3c60bf3faa777c4a221649eb"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.336272 1300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:24:58.337244 1316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.337455 1300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:58.337558 1300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "8b0541e5ccc74b63907465503853adb3"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "6c37e9841fe3d011c6144ccd5a929b8b"
server_key_iv: "929e2dbb3c60bf3faa777c4a221649eb"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.337637 1300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:58.357074 1300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:58.359491 1300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:58.359664 1300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:58.360433 1300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:58.360491 1300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.360525 1300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:58.360548 1300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:58.368633 1300 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:45951
I20260810 07:24:58.368746 1449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:45951 every 8 connection(s)
I20260810 07:24:58.369112 1300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:58.371472 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.361002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:56699 (local address 127.24.60.254:45683)
0810 07:24:58.361077 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.361079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.361363 (+ 284us) server_negotiation.cc:407] Connection header received
0810 07:24:58.361981 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.361983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.361999 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.362084 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.363046 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.363954 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.365098 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.365401 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.365487 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.368245 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.368247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.368248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.368257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.369989 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.370508 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.370509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.370509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.370570 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.370789 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.370789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.370790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.370967 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.371024 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.371330 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.371405 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:24:58.371752 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.360999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.24.60.194:56699)
0810 07:24:58.361211 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.361255 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.361871 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.362146 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.362150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.362547 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:58.362930 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.362937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.364063 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.364064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.364987 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.364991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.365515 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.365516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.365725 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.366431 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:58.366443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:58.368113 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:58.370078 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.370084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.370086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.370430 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.370638 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.370639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.370639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.370748 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.371052 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:58.371059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:58.371198 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.371304 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.371482 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":114,"threads_started":1}
I20260810 07:24:58.372289 1450 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45683
I20260810 07:24:58.372380 1450 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:58.372596 1450 heartbeater.cc:511] Master 127.24.60.254:45683 requested a full tablet report, sending...
I20260810 07:24:58.373121 1060 ts_manager.cc:194] Registered new tserver with Master: 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951)
I20260810 07:24:58.373689 1060 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:56699
I20260810 07:24:58.376010 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 1300
I20260810 07:24:58.376140 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:58.376420 24819 external_mini_cluster.cc:1428] Setting key 461dc3ae35c9fa3bec3e66e770b8b1a1
I20260810 07:24:58.379787 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.373458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.24.60.194:51481)
0810 07:24:58.373585 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.373596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.373657 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.373849 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.373850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.374067 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:58.374190 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.374194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.374836 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.374836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.375337 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.375338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.375671 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.375672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.375721 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.376318 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:58.376324 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:58.376763 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:58.378471 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.378473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.378473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.378731 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.379014 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.379015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.379015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.379089 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.379484 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:58.379485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:58.379533 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.379596 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.379696 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260810 07:24:58.379863 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.373515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:51481 (local address 127.24.60.254:45683)
0810 07:24:58.373558 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.373560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.373660 (+ 100us) server_negotiation.cc:407] Connection header received
0810 07:24:58.373717 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.373718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.373728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.373789 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.374281 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.374746 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.375423 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.375605 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.375659 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.376820 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.376821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.376821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.376828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.378395 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.378814 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.378815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.378815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.378894 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.379162 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.379163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.379163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.379384 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.379461 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.379652 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.379750 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab.
Entry for principal oryx/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab.
I20260810 07:24:58.410284 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45683
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:58.500623 1462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:58.500880 1462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:58.500912 1462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:58.502249 1462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:58.502312 1462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:58.502338 1462 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:24:58.502367 1462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:24:58.504097 1462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.60.254:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:24:58.504401 1462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:58.504665 1462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:58.510402 1469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:58.510486 1471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:58.510538 1468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:58.510584 1462 server_base.cc:1034] running on GCE node
I20260810 07:24:58.510923 1462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:58.511197 1462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:58.512378 1462 hybrid_clock.cc:648] HybridClock initialized: now 1786346698512342 us; error 53 us; skew 500 ppm
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:58.515182 1462 init.cc:377] Logged in from keytab as oryx/127.24.60.195@KRBTEST.COM (short username oryx)
I20260810 07:24:58.515671 1462 webserver.cc:460] Webserver started at http://127.24.60.195:43359/ using document root <none> and password file <none>
I20260810 07:24:58.515946 1462 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:58.516022 1462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:58.516201 1462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:58.517589 1462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "60ee73b1103345368e5ad9ba920ecab6"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "c0038b310ddb821f8324ee5eaec45be1"
server_key_iv: "0d0c1cc3f7144bde2e5824b8721e0c8f"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.517972 1462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "60ee73b1103345368e5ad9ba920ecab6"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "c0038b310ddb821f8324ee5eaec45be1"
server_key_iv: "0d0c1cc3f7144bde2e5824b8721e0c8f"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.519456 1462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:24:58.520215 1478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.520398 1462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:58.520490 1462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "60ee73b1103345368e5ad9ba920ecab6"
format_stamp: "Formatted at 2026-08-10 07:24:58 on dist-test-slave-vxj2"
server_key: "c0038b310ddb821f8324ee5eaec45be1"
server_key_iv: "0d0c1cc3f7144bde2e5824b8721e0c8f"
server_key_version: "encryptionkey@0"
I20260810 07:24:58.520582 1462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:58.529793 1462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:58.532061 1462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:58.532258 1462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:58.532989 1462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:58.533046 1462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.533083 1462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:58.533129 1462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:58.541204 1462 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:36709
I20260810 07:24:58.541282 1613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:36709 every 8 connection(s)
I20260810 07:24:58.541693 1462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:58.544378 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.533520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:55863 (local address 127.24.60.254:45683)
0810 07:24:58.533594 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.533597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.533860 (+ 263us) server_negotiation.cc:407] Connection header received
0810 07:24:58.534720 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.534722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.534737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.534809 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.535815 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.536709 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.537945 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.538162 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.538235 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.540919 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.540921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.540921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.540930 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.542776 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.543269 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.543270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.543270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.543335 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.543594 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.543596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.543596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.543742 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.543890 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.544202 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.544285 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:24:58.544451 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.533491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.24.60.195:55863)
0810 07:24:58.533695 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.533751 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.534549 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.534928 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.534934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.535371 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:58.535701 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.535709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.536892 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.536893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.537835 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.537838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.538279 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.538280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.538400 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.539120 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:58.539128 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:58.540758 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:58.542922 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.542929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.542931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.543184 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.543404 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.543404 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.543404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.543510 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.543911 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:58.543917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:58.544045 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.544139 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.544276 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
I20260810 07:24:58.545025 1614 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45683
I20260810 07:24:58.545125 1614 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:58.545387 1614 heartbeater.cc:511] Master 127.24.60.254:45683 requested a full tablet report, sending...
I20260810 07:24:58.545826 1059 ts_manager.cc:194] Registered new tserver with Master: 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:58.546279 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 1462
I20260810 07:24:58.546367 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:58.546404 1059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:55863
I20260810 07:24:58.546602 24819 external_mini_cluster.cc:1428] Setting key ea29a11b27f1a835a90ec47484ee71cb
I20260810 07:24:58.547560 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:24:58.553576 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.546692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.24.60.195:59671)
0810 07:24:58.547433 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.547455 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.547541 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.547698 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.547699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.547881 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:58.547934 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.547944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.548612 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.548613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.549201 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.549204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.549727 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.549728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.549791 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.550312 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:58.550317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:58.550760 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:58.552417 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.552418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.552418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.552661 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.552884 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:58.552885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:58.552885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:58.552961 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:58.553279 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:58.553280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:58.553325 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.553413 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.553503 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":79,"threads_started":1}
I20260810 07:24:58.553704 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.546977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:59671 (local address 127.24.60.254:45683)
0810 07:24:58.547033 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.547035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.547524 (+ 489us) server_negotiation.cc:407] Connection header received
0810 07:24:58.547592 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.547592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.547603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.547655 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.548036 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.548518 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.549303 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.549641 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.549713 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.550850 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.550851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.550851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.550858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.552331 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.552761 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.552762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.552762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.552816 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.553029 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.553029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.553030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.553209 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.553287 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.553493 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.553580 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:24:58.573325 1112 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:58.566933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.24.60.254:45683)
0810 07:24:58.567001 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.567003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.567018 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:24:58.567046 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.567047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.567056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.567115 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.567791 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.568379 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.569231 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.569437 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.569499 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.573198 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46244: BlockingRecv error: recv got EOF from 127.0.0.1:46244 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 07:24:58 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:58.582017 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.574145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.24.60.254:45683)
0810 07:24:58.574226 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.574229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.574279 (+ 50us) server_negotiation.cc:407] Connection header received
0810 07:24:58.574321 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.574322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.574331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.574403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:58.574879 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.575378 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.576105 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.576343 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.576419 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.579230 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:58.579232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:58.579232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:58.579240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:58.580807 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.581302 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.581302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.581303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.581359 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:58.581556 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:58.581556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:58.581557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:58.581682 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:58.581759 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.581878 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.581946 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:24:58.584091 1059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:58.585840 1059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:58.592883 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.588546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36709 (local address 127.0.0.1:44130)
0810 07:24:58.588812 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.588861 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.588937 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.589205 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.589207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.589209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.589287 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.589289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.590391 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.590393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.591566 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.591569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.592486 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.592488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.592577 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.592593 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.592685 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.592749 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":87,"threads_started":1}
I20260810 07:24:58.592872 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.588684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.24.60.195:36709)
0810 07:24:58.588900 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.588905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.588917 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:24:58.589046 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.589048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.589067 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.589172 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.589422 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.590289 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.591669 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.592390 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.592472 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.592558 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.592707 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.592752 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260810 07:24:58.593145 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.588784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.0.0.1:46032)
0810 07:24:58.589054 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.589063 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.589106 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.589532 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.589532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.589534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.589587 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.589588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.590930 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.590931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.591860 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.591863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.592854 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.592856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.592929 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.592934 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.592989 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.593034 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
I20260810 07:24:58.593333 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.588618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:45951 (local address 127.0.0.1:49448)
0810 07:24:58.589044 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.589056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.589114 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.589466 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.589467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.589469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.589530 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.589531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.590946 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.590947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.592061 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.592064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.593098 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.593099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.593181 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.593185 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.593234 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.593275 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":31,"thread_start_us":93,"threads_started":1}
I20260810 07:24:58.593498 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.589036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.24.60.193:45077)
0810 07:24:58.589298 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.589301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.589309 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:58.589345 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.589347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.589360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.589461 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.589667 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.590814 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.591948 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.592670 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.592745 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.592816 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.593289 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.593331 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260810 07:24:58.594008 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.588763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.24.60.194:45951)
0810 07:24:58.589267 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.589271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.589282 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:24:58.589327 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.589329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.589344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.589612 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.589684 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.590815 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.592157 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.592910 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.592988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.593554 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.593815 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.593875 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":96,"threads_started":1}
I20260810 07:24:58.594017 1537 tablet_service.cc:1467] Processing CreateTablet for tablet a1e9d984bf7b4551a936313d484dc1bb (DEFAULT_TABLE table=test-table [id=9e8d2c269486447daff527145b4bea6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:58.594200 1215 tablet_service.cc:1467] Processing CreateTablet for tablet a1e9d984bf7b4551a936313d484dc1bb (DEFAULT_TABLE table=test-table [id=9e8d2c269486447daff527145b4bea6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:58.594466 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e9d984bf7b4551a936313d484dc1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:58.594502 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e9d984bf7b4551a936313d484dc1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:58.594645 1373 tablet_service.cc:1467] Processing CreateTablet for tablet a1e9d984bf7b4551a936313d484dc1bb (DEFAULT_TABLE table=test-table [id=9e8d2c269486447daff527145b4bea6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:58.595016 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e9d984bf7b4551a936313d484dc1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:58.598081 1645 tablet_bootstrap.cc:492] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Bootstrap starting.
I20260810 07:24:58.598682 1645 tablet_bootstrap.cc:654] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:58.599071 1645 log.cc:824] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:58.599331 1647 tablet_bootstrap.cc:492] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Bootstrap starting.
I20260810 07:24:58.599726 1646 tablet_bootstrap.cc:492] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Bootstrap starting.
I20260810 07:24:58.599985 1645 tablet_bootstrap.cc:492] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: No bootstrap required, opened a new log
I20260810 07:24:58.600077 1645 ts_tablet_manager.cc:1397] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:58.600152 1647 tablet_bootstrap.cc:654] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:58.600621 1646 tablet_bootstrap.cc:654] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:58.600622 1647 log.cc:824] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:58.600986 1646 log.cc:824] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:58.601574 1647 tablet_bootstrap.cc:492] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: No bootstrap required, opened a new log
I20260810 07:24:58.601706 1647 ts_tablet_manager.cc:1397] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:24:58.601881 1646 tablet_bootstrap.cc:492] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: No bootstrap required, opened a new log
I20260810 07:24:58.601953 1646 ts_tablet_manager.cc:1397] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 07:24:58.602262 1645 raft_consensus.cc:348] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.602439 1645 raft_consensus.cc:374] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:58.602494 1645 raft_consensus.cc:729] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Initialized, Role: FOLLOWER
I20260810 07:24:58.602622 1645 consensus_queue.cc:260] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.602936 1645 ts_tablet_manager.cc:1428] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:58.603076 1291 heartbeater.cc:503] Master 127.24.60.254:45683 was elected leader, sending a full tablet report...
I20260810 07:24:58.604045 1646 raft_consensus.cc:348] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.604045 1647 raft_consensus.cc:348] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.604224 1646 raft_consensus.cc:374] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:58.604224 1647 raft_consensus.cc:374] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:58.604264 1646 raft_consensus.cc:729] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Initialized, Role: FOLLOWER
I20260810 07:24:58.604264 1647 raft_consensus.cc:729] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0541e5ccc74b63907465503853adb3, State: Initialized, Role: FOLLOWER
I20260810 07:24:58.604390 1647 consensus_queue.cc:260] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.604390 1646 consensus_queue.cc:260] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.604694 1647 ts_tablet_manager.cc:1428] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:58.604694 1646 ts_tablet_manager.cc:1428] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 07:24:58.604738 1450 heartbeater.cc:503] Master 127.24.60.254:45683 was elected leader, sending a full tablet report...
I20260810 07:24:58.604817 1614 heartbeater.cc:503] Master 127.24.60.254:45683 was elected leader, sending a full tablet report...
W20260810 07:24:58.619837 1451 tablet.cc:2369] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:58.666023 1655 raft_consensus.cc:482] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:58.666215 1655 raft_consensus.cc:504] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.666731 1655 leader_election.cc:288] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077), 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951)
I20260810 07:24:58.669880 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.667012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:46525 (local address 127.24.60.194:45951)
0810 07:24:58.667062 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.667064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.667072 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:58.667103 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.667104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.667113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.667168 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.667330 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.667955 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.668997 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.669476 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.669563 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.669573 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.669750 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.669793 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 07:24:58.669888 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.666911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:45951 (local address 127.24.60.195:46525)
0810 07:24:58.666971 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.666984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.667031 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.667173 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.667173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.667175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.667249 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.667252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.668070 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.668071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.668892 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.668894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.669561 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.669562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.669674 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.669677 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.669745 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.669794 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 07:24:58.669988 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.666987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:49599 (local address 127.24.60.193:45077)
0810 07:24:58.667041 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.667043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.667202 (+ 159us) server_negotiation.cc:407] Connection header received
0810 07:24:58.667274 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.667275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.667285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.667357 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.667533 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.668178 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.669148 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.669588 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.669673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.669683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.669845 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.669895 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 07:24:58.670035 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.666929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.24.60.195:49599)
0810 07:24:58.667119 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.667129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.667217 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.667368 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.667370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.667371 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.667443 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.667445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.668271 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.668273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.669045 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.669047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.669674 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.669675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.669752 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.669756 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.669836 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.669880 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":130,"threads_started":1}
I20260810 07:24:58.670308 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e9d984bf7b4551a936313d484dc1bb" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3" is_pre_election: true
I20260810 07:24:58.670408 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e9d984bf7b4551a936313d484dc1bb" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" is_pre_election: true
I20260810 07:24:58.670490 1397 raft_consensus.cc:2454] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 0.
I20260810 07:24:58.670604 1235 raft_consensus.cc:2454] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 0.
I20260810 07:24:58.670771 1482 leader_election.cc:302] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ee73b1103345368e5ad9ba920ecab6, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:58.670971 1655 raft_consensus.cc:2781] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:58.671052 1655 raft_consensus.cc:482] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:58.671088 1655 raft_consensus.cc:3037] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:58.672159 1655 raft_consensus.cc:504] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.672334 1655 leader_election.cc:288] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 election: Requested vote from peers 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077), 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951)
I20260810 07:24:58.672555 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e9d984bf7b4551a936313d484dc1bb" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a8cd4ab84b49e49daa7ec296c646d7"
I20260810 07:24:58.672556 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e9d984bf7b4551a936313d484dc1bb" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3"
I20260810 07:24:58.672663 1397 raft_consensus.cc:3037] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:58.672665 1235 raft_consensus.cc:3037] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:58.673596 1235 raft_consensus.cc:2454] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 1.
I20260810 07:24:58.673588 1397 raft_consensus.cc:2454] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 1.
I20260810 07:24:58.673825 1480 leader_election.cc:302] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 60ee73b1103345368e5ad9ba920ecab6; no voters:
I20260810 07:24:58.673943 1655 raft_consensus.cc:2781] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:58.674216 1655 raft_consensus.cc:686] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 1 LEADER]: Becoming Leader. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Running, Role: LEADER
I20260810 07:24:58.674337 1655 consensus_queue.cc:237] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } }
I20260810 07:24:58.675392 1058 catalog_manager.cc:5495] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195). New cstate: current_term: 1 leader_uuid: "60ee73b1103345368e5ad9ba920ecab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:58.690330 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.687514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.24.60.195:36709)
0810 07:24:58.687587 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.687589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.687597 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:24:58.687635 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.687636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.687656 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.687732 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:58.688017 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.688494 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.689501 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.689700 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.689753 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.689991 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:58.690103 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:58.690168 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.690247 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.690267 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 07:24:58.691725 1292 tablet.cc:2369] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:58.692513 1235 raft_consensus.cc:1261] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Refusing update from remote peer 60ee73b1103345368e5ad9ba920ecab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:58.692579 1397 raft_consensus.cc:1261] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Refusing update from remote peer 60ee73b1103345368e5ad9ba920ecab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:58.692843 1655 consensus_queue.cc:1035] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:58.693011 1657 consensus_queue.cc:1035] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:58.695782 1660 mvcc.cc:205] Tried to move back new op lower bound from 7316876077841416192 to 7316876077770891264. Current Snapshot: MvccSnapshot[applied={T|T < 7316876077841416192}]
I20260810 07:24:58.696106 1662 mvcc.cc:205] Tried to move back new op lower bound from 7316876077841416192 to 7316876077770891264. Current Snapshot: MvccSnapshot[applied={T|T < 7316876077841416192}]
I20260810 07:24:58.696472 1663 mvcc.cc:205] Tried to move back new op lower bound from 7316876077841416192 to 7316876077770891264. Current Snapshot: MvccSnapshot[applied={T|T < 7316876077841416192}]
I20260810 07:24:58.698550 1059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:58.698638 1059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:58.699386 1059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Sending DeleteTablet for 3 replicas of tablet a1e9d984bf7b4551a936313d484dc1bb
I20260810 07:24:58.699892 1373 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e9d984bf7b4551a936313d484dc1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:58 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49448
I20260810 07:24:58.699934 1537 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e9d984bf7b4551a936313d484dc1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:58 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:44130
I20260810 07:24:58.699928 1215 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e9d984bf7b4551a936313d484dc1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:58 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:46032
I20260810 07:24:58.700317 1673 tablet_replica.cc:330] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: stopping tablet replica
I20260810 07:24:58.700466 1673 raft_consensus.cc:2229] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:58.700402 1059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:58.700588 1059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:58.700599 1673 raft_consensus.cc:2258] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:58.700942 1671 tablet_replica.cc:330] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: stopping tablet replica
I20260810 07:24:58.701049 1671 raft_consensus.cc:2229] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:58.701205 1672 tablet_replica.cc:330] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: stopping tablet replica
I20260810 07:24:58.701222 1671 raft_consensus.cc:2258] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:58.701452 1672 raft_consensus.cc:2229] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:58.701629 1672 raft_consensus.cc:2258] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:58.701714 1671 ts_tablet_manager.cc:1905] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:58.702265 1672 ts_tablet_manager.cc:1905] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:58.702399 1373 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4d69eef47e45ecb16445e9efd230d5 (DEFAULT_TABLE table=test-table [id=2681a08e733645278146b11592c1ad18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:58.702579 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d69eef47e45ecb16445e9efd230d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:58.702780 1215 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4d69eef47e45ecb16445e9efd230d5 (DEFAULT_TABLE table=test-table [id=2681a08e733645278146b11592c1ad18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:58.702911 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d69eef47e45ecb16445e9efd230d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:58.703938 1671 ts_tablet_manager.cc:1918] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:58.704012 1671 log.cc:1196] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/a1e9d984bf7b4551a936313d484dc1bb
I20260810 07:24:58.704134 1647 tablet_bootstrap.cc:492] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Bootstrap starting.
I20260810 07:24:58.704360 1645 tablet_bootstrap.cc:492] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Bootstrap starting.
I20260810 07:24:58.704643 1647 tablet_bootstrap.cc:654] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:58.704806 1645 tablet_bootstrap.cc:654] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:58.704304 1671 ts_tablet_manager.cc:1939] T a1e9d984bf7b4551a936313d484dc1bb P 60ee73b1103345368e5ad9ba920ecab6: Deleting consensus metadata
I20260810 07:24:58.704437 1537 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4d69eef47e45ecb16445e9efd230d5 (DEFAULT_TABLE table=test-table [id=2681a08e733645278146b11592c1ad18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:58.705335 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d69eef47e45ecb16445e9efd230d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:58.705894 1645 tablet_bootstrap.cc:492] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: No bootstrap required, opened a new log
I20260810 07:24:58.705955 1645 ts_tablet_manager.cc:1397] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:58.706384 1672 ts_tablet_manager.cc:1918] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:58.706382 1645 raft_consensus.cc:348] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.706475 1645 raft_consensus.cc:374] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:58.706501 1645 raft_consensus.cc:729] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Initialized, Role: FOLLOWER
I20260810 07:24:58.706571 1645 consensus_queue.cc:260] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.706674 1672 log.cc:1196] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/a1e9d984bf7b4551a936313d484dc1bb
I20260810 07:24:58.706682 1645 ts_tablet_manager.cc:1428] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:58.706686 1646 tablet_bootstrap.cc:492] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Bootstrap starting.
I20260810 07:24:58.707086 1672 ts_tablet_manager.cc:1939] T a1e9d984bf7b4551a936313d484dc1bb P 8b0541e5ccc74b63907465503853adb3: Deleting consensus metadata
I20260810 07:24:58.707127 1646 tablet_bootstrap.cc:654] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:58.707579 1040 catalog_manager.cc:4841] TS 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951): tablet a1e9d984bf7b4551a936313d484dc1bb (table test-table [id=9e8d2c269486447daff527145b4bea6d]) successfully deleted
I20260810 07:24:58.707953 1038 catalog_manager.cc:4841] TS 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709): tablet a1e9d984bf7b4551a936313d484dc1bb (table test-table [id=9e8d2c269486447daff527145b4bea6d]) successfully deleted
I20260810 07:24:58.708313 1646 tablet_bootstrap.cc:492] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: No bootstrap required, opened a new log
I20260810 07:24:58.708375 1646 ts_tablet_manager.cc:1397] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 07:24:58.708521 1646 raft_consensus.cc:348] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.708609 1646 raft_consensus.cc:374] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:58.708631 1646 raft_consensus.cc:729] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Initialized, Role: FOLLOWER
I20260810 07:24:58.708676 1646 consensus_queue.cc:260] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.708774 1646 ts_tablet_manager.cc:1428] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 07:24:58.709173 1673 ts_tablet_manager.cc:1905] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:58.709673 1647 tablet_bootstrap.cc:492] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: No bootstrap required, opened a new log
I20260810 07:24:58.709726 1647 ts_tablet_manager.cc:1397] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 07:24:58.709892 1647 raft_consensus.cc:348] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.709967 1647 raft_consensus.cc:374] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:58.709996 1647 raft_consensus.cc:729] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0541e5ccc74b63907465503853adb3, State: Initialized, Role: FOLLOWER
I20260810 07:24:58.710049 1647 consensus_queue.cc:260] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.710179 1647 ts_tablet_manager.cc:1428] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:58.711000 1673 ts_tablet_manager.cc:1918] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:58.711098 1673 log.cc:1196] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/a1e9d984bf7b4551a936313d484dc1bb
I20260810 07:24:58.711380 1673 ts_tablet_manager.cc:1939] T a1e9d984bf7b4551a936313d484dc1bb P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting consensus metadata
I20260810 07:24:58.711894 1038 catalog_manager.cc:4841] TS 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077): tablet a1e9d984bf7b4551a936313d484dc1bb (table test-table [id=9e8d2c269486447daff527145b4bea6d]) successfully deleted
W20260810 07:24:58.792557 1615 tablet.cc:2369] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:24:58.985986 1654 raft_consensus.cc:482] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:58.986141 1654 raft_consensus.cc:504] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.986560 1654 leader_election.cc:288] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077), 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:58.990125 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.986819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.24.60.194:59931)
0810 07:24:58.987080 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.987096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.987214 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.987495 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.987496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.987498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.987568 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.987570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.988289 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.988290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.989181 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.989184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.989799 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.989800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.989885 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.989890 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.989944 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.990012 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260810 07:24:58.990134 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.987040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:59931 (local address 127.24.60.193:45077)
0810 07:24:58.987101 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.987103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.987178 (+ 75us) server_negotiation.cc:407] Connection header received
0810 07:24:58.987377 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.987378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.987386 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.987465 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.987687 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.988181 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.989300 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.989695 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.989751 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.989758 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.990007 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.990060 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:24:58.990231 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.987120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:50045 (local address 127.24.60.195:36709)
0810 07:24:58.987179 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:58.987182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:58.987232 (+ 50us) server_negotiation.cc:407] Connection header received
0810 07:24:58.987344 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:58.987345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:58.987360 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:58.987419 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:58.987631 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.988367 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.989376 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.989790 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.989869 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.989879 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:58.990100 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:24:58.990159 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 07:24:58.990299 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:58.987044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36709 (local address 127.24.60.194:50045)
0810 07:24:58.987169 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:58.987187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:58.987240 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:58.987457 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:58.987458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:58.987462 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:58.987551 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.987553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.988479 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.988480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.989289 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:58.989292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:58.989891 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:58.989892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:58.990015 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:58.990020 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:24:58.990078 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 07:24:58.990119 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":48,"thread_start_us":41,"threads_started":1}
I20260810 07:24:58.990479 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d69eef47e45ecb16445e9efd230d5" candidate_uuid: "8b0541e5ccc74b63907465503853adb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" is_pre_election: true
I20260810 07:24:58.990571 1235 raft_consensus.cc:2454] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0541e5ccc74b63907465503853adb3 in term 0.
I20260810 07:24:58.990775 1317 leader_election.cc:302] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:58.990924 1654 raft_consensus.cc:2781] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:58.990823 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d69eef47e45ecb16445e9efd230d5" candidate_uuid: "8b0541e5ccc74b63907465503853adb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ee73b1103345368e5ad9ba920ecab6" is_pre_election: true
I20260810 07:24:58.990967 1654 raft_consensus.cc:482] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:58.991021 1654 raft_consensus.cc:3037] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:58.991001 1558 raft_consensus.cc:2454] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0541e5ccc74b63907465503853adb3 in term 0.
I20260810 07:24:58.991884 1654 raft_consensus.cc:504] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.992049 1654 leader_election.cc:288] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [CANDIDATE]: Term 1 election: Requested vote from peers 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077), 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:58.992242 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d69eef47e45ecb16445e9efd230d5" candidate_uuid: "8b0541e5ccc74b63907465503853adb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a8cd4ab84b49e49daa7ec296c646d7"
I20260810 07:24:58.992331 1235 raft_consensus.cc:3037] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:58.992321 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d69eef47e45ecb16445e9efd230d5" candidate_uuid: "8b0541e5ccc74b63907465503853adb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ee73b1103345368e5ad9ba920ecab6"
I20260810 07:24:58.992413 1558 raft_consensus.cc:3037] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:58.993130 1235 raft_consensus.cc:2454] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0541e5ccc74b63907465503853adb3 in term 1.
I20260810 07:24:58.993240 1558 raft_consensus.cc:2454] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0541e5ccc74b63907465503853adb3 in term 1.
I20260810 07:24:58.993367 1317 leader_election.cc:302] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:58.993496 1654 raft_consensus.cc:2781] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:58.993656 1654 raft_consensus.cc:686] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0541e5ccc74b63907465503853adb3, State: Running, Role: LEADER
I20260810 07:24:58.993737 1654 consensus_queue.cc:237] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:58.994568 1059 catalog_manager.cc:5495] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0541e5ccc74b63907465503853adb3 (127.24.60.194). New cstate: current_term: 1 leader_uuid: "8b0541e5ccc74b63907465503853adb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } health_report { overall_health: UNKNOWN } } }
Aug 10 07:24:59 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.254@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:59.085561 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.078069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45683 (local address 127.0.0.1:46260)
0810 07:24:59.078268 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.078280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.078352 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.078509 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.078510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.078830 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:59.078883 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.078885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.079576 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.079577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.080139 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.080141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.080512 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.080513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.080558 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.081139 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:59.081144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:59.082785 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:59.084504 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:59.084512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:59.084514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:59.084726 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:59.084915 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:59.084915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:59.084916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:59.084983 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:59.085233 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:59.085245 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:59.085292 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.085349 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.085462 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260810 07:24:59.085637 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.078181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.24.60.254:45683)
0810 07:24:59.078287 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.078289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.078354 (+ 65us) server_negotiation.cc:407] Connection header received
0810 07:24:59.078404 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.078404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.078418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.078484 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:59.078959 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.079495 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.080229 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.080436 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.080493 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.082877 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:59.082879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:59.082879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:59.082888 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:59.084422 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.084793 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.084794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.084794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.084851 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.085037 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.085038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.085038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.085175 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:59.085227 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.085410 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.085512 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":72}
I20260810 07:24:59.087643 1059 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:46260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 07:24:59.087904 1059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:59.089699 1373 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa63bbf5f1340fea4e90eb4bbb8811e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=428330c467844b94b871d9b6271eb410]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:59.089694 1215 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa63bbf5f1340fea4e90eb4bbb8811e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=428330c467844b94b871d9b6271eb410]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:59.089716 1537 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa63bbf5f1340fea4e90eb4bbb8811e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=428330c467844b94b871d9b6271eb410]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:24:59.089903 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa63bbf5f1340fea4e90eb4bbb8811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.089908 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa63bbf5f1340fea4e90eb4bbb8811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.089903 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa63bbf5f1340fea4e90eb4bbb8811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.091563 1645 tablet_bootstrap.cc:492] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: Bootstrap starting.
I20260810 07:24:59.092033 1645 tablet_bootstrap.cc:654] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.092437 1647 tablet_bootstrap.cc:492] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: Bootstrap starting.
I20260810 07:24:59.092810 1645 tablet_bootstrap.cc:492] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: No bootstrap required, opened a new log
I20260810 07:24:59.092873 1645 ts_tablet_manager.cc:1397] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:59.092976 1647 tablet_bootstrap.cc:654] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.093057 1645 raft_consensus.cc:348] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.093209 1645 raft_consensus.cc:374] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.093240 1645 raft_consensus.cc:729] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.093253 1646 tablet_bootstrap.cc:492] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: Bootstrap starting.
I20260810 07:24:59.093302 1645 consensus_queue.cc:260] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.093461 1645 ts_tablet_manager.cc:1428] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:24:59.093734 1646 tablet_bootstrap.cc:654] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.093672 1652 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.093786 1652 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.094086 1647 tablet_bootstrap.cc:492] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: No bootstrap required, opened a new log
I20260810 07:24:59.094136 1647 ts_tablet_manager.cc:1397] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:59.094305 1647 raft_consensus.cc:348] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.094426 1647 raft_consensus.cc:374] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.094453 1647 raft_consensus.cc:729] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0541e5ccc74b63907465503853adb3, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.094502 1647 consensus_queue.cc:260] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.094594 1646 tablet_bootstrap.cc:492] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: No bootstrap required, opened a new log
I20260810 07:24:59.094615 1647 ts_tablet_manager.cc:1428] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.094642 1646 ts_tablet_manager.cc:1397] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:59.094684 1654 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.094751 1646 raft_consensus.cc:348] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.094789 1654 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.094826 1646 raft_consensus.cc:374] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.094851 1646 raft_consensus.cc:729] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.094898 1646 consensus_queue.cc:260] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.094980 1646 ts_tablet_manager.cc:1428] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.095082 1657 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.095199 1657 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.268285 1652 raft_consensus.cc:482] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:59.268437 1652 raft_consensus.cc:504] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.268776 1652 leader_election.cc:288] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951), 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:59.272405 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.269018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36709 (local address 127.24.60.193:58643)
0810 07:24:59.269263 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.269275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.269368 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.269537 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.269538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.269541 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:59.269707 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.269710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.270447 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.270448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.271389 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.271392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.272030 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.272031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.272110 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.272116 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.272220 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.272272 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":158,"thread_start_us":71,"threads_started":1}
I20260810 07:24:59.272385 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.269091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58643 (local address 127.24.60.195:36709)
0810 07:24:59.269190 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.269192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.269383 (+ 191us) server_negotiation.cc:407] Connection header received
0810 07:24:59.269430 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.269431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.269441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.269521 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:59.269832 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.270314 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.271502 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.271956 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.272017 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.272025 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.272228 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.272281 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:24:59.272496 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.269190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:54885 (local address 127.24.60.194:45951)
0810 07:24:59.269312 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.269314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.269324 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:59.269405 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.269406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.269418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.269485 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:24:59.269770 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.270301 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.271542 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.272082 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.272163 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.272173 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.272360 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.272414 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260810 07:24:59.272660 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.269018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:45951 (local address 127.24.60.193:54885)
0810 07:24:59.269238 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.269256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.269312 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.269512 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.269513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.269516 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:24:59.269619 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.269621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.270417 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.270419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.271410 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.271413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.272194 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.272195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.272272 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.272276 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.272337 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.272379 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":74,"thread_start_us":73,"threads_started":1}
I20260810 07:24:59.273046 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa63bbf5f1340fea4e90eb4bbb8811e" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ee73b1103345368e5ad9ba920ecab6" is_pre_election: true
I20260810 07:24:59.273070 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa63bbf5f1340fea4e90eb4bbb8811e" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3" is_pre_election: true
I20260810 07:24:59.273209 1558 raft_consensus.cc:2454] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 0.
I20260810 07:24:59.273206 1397 raft_consensus.cc:2454] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 0.
I20260810 07:24:59.273511 1160 leader_election.cc:302] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:59.273695 1652 raft_consensus.cc:2781] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:59.273764 1652 raft_consensus.cc:482] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:59.273789 1652 raft_consensus.cc:3037] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.274611 1652 raft_consensus.cc:504] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.274775 1652 leader_election.cc:288] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951), 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:59.274989 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa63bbf5f1340fea4e90eb4bbb8811e" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ee73b1103345368e5ad9ba920ecab6"
I20260810 07:24:59.274979 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa63bbf5f1340fea4e90eb4bbb8811e" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3"
I20260810 07:24:59.275072 1397 raft_consensus.cc:3037] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.275072 1558 raft_consensus.cc:3037] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.275905 1558 raft_consensus.cc:2454] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 1.
I20260810 07:24:59.275905 1397 raft_consensus.cc:2454] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 1.
I20260810 07:24:59.276108 1160 leader_election.cc:302] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:59.276268 1652 raft_consensus.cc:2781] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:59.276386 1652 raft_consensus.cc:686] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 LEADER]: Becoming Leader. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Running, Role: LEADER
I20260810 07:24:59.276473 1652 consensus_queue.cc:237] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.276841 1694 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: TxnStatusTablet state changed. Reason: New leader 29a8cd4ab84b49e49daa7ec296c646d7. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.276984 1694 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:59.277279 1059 catalog_manager.cc:5495] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:59.277602 1696 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 07:24:59.277724 1696 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 07:24:59.323199 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.320410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.0.0.1:46036)
0810 07:24:59.320506 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.320534 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.320588 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.320781 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.320783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.320784 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:59.320843 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.320845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.321534 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.321535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.322136 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.322138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.322498 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.322499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.322558 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.322579 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.322997 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.323009 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.323050 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.323099 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 07:24:59.323213 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.320493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.24.60.193:45077)
0810 07:24:59.320565 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.320568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.320578 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:59.320687 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.320688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.320707 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.320774 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.320917 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.321452 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.322236 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.322425 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.322498 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.322744 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.322881 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.322980 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.323103 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.323131 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:24:59.324632 1397 raft_consensus.cc:1261] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Refusing update from remote peer 29a8cd4ab84b49e49daa7ec296c646d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.324676 1558 raft_consensus.cc:1261] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Refusing update from remote peer 29a8cd4ab84b49e49daa7ec296c646d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.324916 1694 consensus_queue.cc:1035] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.325093 1652 consensus_queue.cc:1035] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.325886 1654 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: TxnStatusTablet state changed. Reason: New leader 29a8cd4ab84b49e49daa7ec296c646d7. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.326027 1654 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.326040 1657 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: TxnStatusTablet state changed. Reason: New leader 29a8cd4ab84b49e49daa7ec296c646d7. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.326150 1657 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.327255 1652 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.327349 1652 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:59.327399 1652 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.327437 1652 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 29a8cd4ab84b49e49daa7ec296c646d7: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:24:59.327461 1654 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.327553 1654 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 8b0541e5ccc74b63907465503853adb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.327607 1657 tablet_replica.cc:439] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } } }
I20260810 07:24:59.327692 1657 tablet_replica.cc:442] T 1fa63bbf5f1340fea4e90eb4bbb8811e P 60ee73b1103345368e5ad9ba920ecab6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:24:59.332329 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.329738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.24.60.194:45951)
0810 07:24:59.329799 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.329802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.329835 (+ 33us) server_negotiation.cc:407] Connection header received
0810 07:24:59.329933 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.329934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.329954 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.330055 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.330216 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.330706 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.331485 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.331701 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.331775 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.331958 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.332075 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.332163 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.332211 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.332235 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:24:59.338351 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.335373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:38987 (local address 127.24.60.193:45077)
0810 07:24:59.335448 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.335451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.335628 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:24:59.335694 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.335695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.335705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.335754 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.335893 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.336367 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.337493 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.337711 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.337863 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.338078 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.338118 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.338169 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.338257 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.338278 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:24:59.338425 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.335333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.24.60.194:38987)
0810 07:24:59.335549 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.335562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.335632 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.335759 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.335760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.335762 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:59.335830 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.335832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.336459 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.336460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.337368 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.337372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.337830 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.337831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.337895 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.337915 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.338191 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.338194 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.338250 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.338312 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":118,"threads_started":1}
I20260810 07:24:59.341544 1558 raft_consensus.cc:1261] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Refusing update from remote peer 8b0541e5ccc74b63907465503853adb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.341559 1235 raft_consensus.cc:1261] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Refusing update from remote peer 8b0541e5ccc74b63907465503853adb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.341867 1654 consensus_queue.cc:1035] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.342069 1679 consensus_queue.cc:1035] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.356443 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.353871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53657 (local address 127.24.60.194:45951)
0810 07:24:59.353954 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.353957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.354000 (+ 43us) server_negotiation.cc:407] Connection header received
0810 07:24:59.354078 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.354080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.354094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.354143 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.354301 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.354793 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.355611 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.355836 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.355926 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.356129 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.356168 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.356215 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.356310 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.356350 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:24:59.356735 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.353666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:45951 (local address 127.24.60.193:53657)
0810 07:24:59.353940 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.353954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.354017 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.354144 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.354145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.354146 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:59.354220 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.354222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.354887 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.354888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.355518 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.355520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.355924 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.355925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.356013 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.356031 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.356238 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.356241 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.356596 (+ 355us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.356663 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":124,"threads_started":1}
I20260810 07:24:59.367465 1059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:59.367600 1059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:59.368433 1059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Sending DeleteTablet for 3 replicas of tablet 2d4d69eef47e45ecb16445e9efd230d5
I20260810 07:24:59.368722 1373 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4d69eef47e45ecb16445e9efd230d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49448
I20260810 07:24:59.369028 1215 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4d69eef47e45ecb16445e9efd230d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:46032
I20260810 07:24:59.369086 1537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4d69eef47e45ecb16445e9efd230d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:44130
I20260810 07:24:59.377306 1725 tablet_replica.cc:330] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: stopping tablet replica
I20260810 07:24:59.377447 1725 raft_consensus.cc:2229] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:59.377506 1725 raft_consensus.cc:2258] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.377667 1724 tablet_replica.cc:330] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: stopping tablet replica
I20260810 07:24:59.377735 1724 raft_consensus.cc:2229] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:59.377789 1724 raft_consensus.cc:2258] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.377787 1723 tablet_replica.cc:330] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: stopping tablet replica
I20260810 07:24:59.377871 1723 raft_consensus.cc:2229] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:59.378034 1723 raft_consensus.cc:2258] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.378083 1725 ts_tablet_manager.cc:1905] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:59.378433 1723 ts_tablet_manager.cc:1905] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:59.380426 1723 ts_tablet_manager.cc:1918] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:59.380410 1724 ts_tablet_manager.cc:1905] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:59.380601 1723 log.cc:1196] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/2d4d69eef47e45ecb16445e9efd230d5
I20260810 07:24:59.380971 1723 ts_tablet_manager.cc:1939] T 2d4d69eef47e45ecb16445e9efd230d5 P 8b0541e5ccc74b63907465503853adb3: Deleting consensus metadata
I20260810 07:24:59.381515 1040 catalog_manager.cc:4841] TS 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951): tablet 2d4d69eef47e45ecb16445e9efd230d5 (table test-table [id=2681a08e733645278146b11592c1ad18]) successfully deleted
I20260810 07:24:59.382666 1724 ts_tablet_manager.cc:1918] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:59.382763 1724 log.cc:1196] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/2d4d69eef47e45ecb16445e9efd230d5
I20260810 07:24:59.383033 1724 ts_tablet_manager.cc:1939] T 2d4d69eef47e45ecb16445e9efd230d5 P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting consensus metadata
I20260810 07:24:59.383261 1725 ts_tablet_manager.cc:1918] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:24:59.383347 1725 log.cc:1196] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/2d4d69eef47e45ecb16445e9efd230d5
I20260810 07:24:59.383600 1725 ts_tablet_manager.cc:1939] T 2d4d69eef47e45ecb16445e9efd230d5 P 60ee73b1103345368e5ad9ba920ecab6: Deleting consensus metadata
I20260810 07:24:59.384052 1038 catalog_manager.cc:4841] TS 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709): tablet 2d4d69eef47e45ecb16445e9efd230d5 (table test-table [id=2681a08e733645278146b11592c1ad18]) successfully deleted
I20260810 07:24:59.384295 1038 catalog_manager.cc:4841] TS 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077): tablet 2d4d69eef47e45ecb16445e9efd230d5 (table test-table [id=2681a08e733645278146b11592c1ad18]) successfully deleted
Aug 10 07:24:59 dist-test-slave-vxj2 krb5kdc[1002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:24:59 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:24:59 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:24:59.417203 1112 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:24:59.409562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.24.60.254:45683)
0810 07:24:59.409619 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.409621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.409838 (+ 217us) server_negotiation.cc:407] Connection header received
0810 07:24:59.409902 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.409904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.409918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.409987 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:59.410868 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.411685 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.412806 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.413135 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.413322 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.417105 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46266: BlockingRecv error: recv got EOF from 127.0.0.1:46266 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 10 07:24:59 dist-test-slave-vxj2 krb5kdc[1002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM
I20260810 07:24:59.428157 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.418439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.24.60.254:45683)
0810 07:24:59.418563 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.418565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.418821 (+ 256us) server_negotiation.cc:407] Connection header received
0810 07:24:59.418879 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.418880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.418892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.418957 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:59.419580 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.420325 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.421462 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.421748 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.421828 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.424835 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:59.424837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:59.424837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:59.424847 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:59.426709 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.427161 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.427162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.427163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.427230 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.427473 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.427474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.427475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.427633 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:59.427740 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.427895 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.428054 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 07:24:59.430084 1059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:24:59.430284 1059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:59.431695 1537 tablet_service.cc:1467] Processing CreateTablet for tablet f28367ff83374e24a2525daea30da3a6 (DEFAULT_TABLE table=test-table [id=5271ca15368b4c91a1c034cec72557c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:59.431802 1215 tablet_service.cc:1467] Processing CreateTablet for tablet f28367ff83374e24a2525daea30da3a6 (DEFAULT_TABLE table=test-table [id=5271ca15368b4c91a1c034cec72557c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:59.431851 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28367ff83374e24a2525daea30da3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.431936 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28367ff83374e24a2525daea30da3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.433269 1645 tablet_bootstrap.cc:492] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Bootstrap starting.
I20260810 07:24:59.433303 1373 tablet_service.cc:1467] Processing CreateTablet for tablet f28367ff83374e24a2525daea30da3a6 (DEFAULT_TABLE table=test-table [id=5271ca15368b4c91a1c034cec72557c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:59.433439 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28367ff83374e24a2525daea30da3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.433717 1645 tablet_bootstrap.cc:654] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.434260 1646 tablet_bootstrap.cc:492] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Bootstrap starting.
I20260810 07:24:59.434643 1646 tablet_bootstrap.cc:654] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.434656 1647 tablet_bootstrap.cc:492] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Bootstrap starting.
I20260810 07:24:59.435096 1647 tablet_bootstrap.cc:654] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.435590 1646 tablet_bootstrap.cc:492] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: No bootstrap required, opened a new log
I20260810 07:24:59.435635 1646 ts_tablet_manager.cc:1397] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:59.435743 1646 raft_consensus.cc:348] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.435822 1646 raft_consensus.cc:374] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.435842 1646 raft_consensus.cc:729] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.435883 1646 consensus_queue.cc:260] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.435978 1646 ts_tablet_manager.cc:1428] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.436414 1647 tablet_bootstrap.cc:492] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: No bootstrap required, opened a new log
I20260810 07:24:59.436474 1647 ts_tablet_manager.cc:1397] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:59.436616 1647 raft_consensus.cc:348] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.436679 1647 raft_consensus.cc:374] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.436697 1647 raft_consensus.cc:729] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0541e5ccc74b63907465503853adb3, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.436735 1647 consensus_queue.cc:260] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.436807 1647 ts_tablet_manager.cc:1428] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.436861 1645 tablet_bootstrap.cc:492] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: No bootstrap required, opened a new log
I20260810 07:24:59.436908 1645 ts_tablet_manager.cc:1397] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 07:24:59.437090 1645 raft_consensus.cc:348] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.437218 1645 raft_consensus.cc:374] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.437242 1645 raft_consensus.cc:729] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.437297 1645 consensus_queue.cc:260] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.437397 1645 ts_tablet_manager.cc:1428] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:59.656430 1705 raft_consensus.cc:482] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:59.656567 1705 raft_consensus.cc:504] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.656786 1705 leader_election.cc:288] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951), 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:59.657168 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28367ff83374e24a2525daea30da3a6" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3" is_pre_election: true
I20260810 07:24:59.657269 1397 raft_consensus.cc:2454] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 0.
I20260810 07:24:59.657143 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28367ff83374e24a2525daea30da3a6" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ee73b1103345368e5ad9ba920ecab6" is_pre_election: true
I20260810 07:24:59.657325 1558 raft_consensus.cc:2454] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 0.
I20260810 07:24:59.657482 1160 leader_election.cc:302] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:59.657603 1705 raft_consensus.cc:2781] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:59.657661 1705 raft_consensus.cc:482] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:59.657696 1705 raft_consensus.cc:3037] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.658558 1705 raft_consensus.cc:504] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.658733 1705 leader_election.cc:288] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951), 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709)
I20260810 07:24:59.658977 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28367ff83374e24a2525daea30da3a6" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3"
I20260810 07:24:59.658975 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28367ff83374e24a2525daea30da3a6" candidate_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ee73b1103345368e5ad9ba920ecab6"
I20260810 07:24:59.659078 1397 raft_consensus.cc:3037] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.659078 1558 raft_consensus.cc:3037] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.659973 1558 raft_consensus.cc:2454] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 1.
I20260810 07:24:59.659979 1397 raft_consensus.cc:2454] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a8cd4ab84b49e49daa7ec296c646d7 in term 1.
I20260810 07:24:59.660183 1160 leader_election.cc:302] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 8b0541e5ccc74b63907465503853adb3; no voters:
I20260810 07:24:59.660320 1705 raft_consensus.cc:2781] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:59.660391 1705 raft_consensus.cc:686] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 LEADER]: Becoming Leader. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Running, Role: LEADER
I20260810 07:24:59.660449 1705 consensus_queue.cc:237] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.661118 1058 catalog_manager.cc:5495] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } health_report { overall_health: UNKNOWN } } }
I20260810 07:24:59.667040 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.664407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.24.60.193:45077)
0810 07:24:59.664471 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.664473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.664483 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:24:59.664556 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.664557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.664569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.664642 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.664760 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.665365 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.666274 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.666518 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.666595 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.666767 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.666810 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.666868 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.666923 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.666948 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:24:59.668143 1558 raft_consensus.cc:1261] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Refusing update from remote peer 29a8cd4ab84b49e49daa7ec296c646d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.668141 1397 raft_consensus.cc:1261] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Refusing update from remote peer 29a8cd4ab84b49e49daa7ec296c646d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.668367 1694 consensus_queue.cc:1035] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.668471 1705 consensus_queue.cc:1035] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.671998 1058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:59.672091 1058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:59.672780 1058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Sending DeleteTablet for 3 replicas of tablet f28367ff83374e24a2525daea30da3a6
I20260810 07:24:59.673069 1215 tablet_service.cc:1514] Processing DeleteTablet for tablet f28367ff83374e24a2525daea30da3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:46032
I20260810 07:24:59.673210 1537 tablet_service.cc:1514] Processing DeleteTablet for tablet f28367ff83374e24a2525daea30da3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:44130
I20260810 07:24:59.673264 1724 tablet_replica.cc:330] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: stopping tablet replica
I20260810 07:24:59.673317 1725 tablet_replica.cc:330] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: stopping tablet replica
I20260810 07:24:59.673331 1724 raft_consensus.cc:2229] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:59.673396 1725 raft_consensus.cc:2229] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:59.673451 1724 raft_consensus.cc:2258] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.673461 1725 raft_consensus.cc:2258] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.673550 1373 tablet_service.cc:1514] Processing DeleteTablet for tablet f28367ff83374e24a2525daea30da3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49448
I20260810 07:24:59.673702 1723 tablet_replica.cc:330] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: stopping tablet replica
I20260810 07:24:59.673758 1723 raft_consensus.cc:2229] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:59.673810 1723 raft_consensus.cc:2258] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.673789 1725 ts_tablet_manager.cc:1905] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:59.674109 1724 ts_tablet_manager.cc:1905] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:59.675688 1725 ts_tablet_manager.cc:1918] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:59.675823 1725 log.cc:1196] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/f28367ff83374e24a2525daea30da3a6
I20260810 07:24:59.675884 1724 ts_tablet_manager.cc:1918] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:59.675961 1724 log.cc:1196] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/f28367ff83374e24a2525daea30da3a6
I20260810 07:24:59.676070 1725 ts_tablet_manager.cc:1939] T f28367ff83374e24a2525daea30da3a6 P 60ee73b1103345368e5ad9ba920ecab6: Deleting consensus metadata
I20260810 07:24:59.676193 1724 ts_tablet_manager.cc:1939] T f28367ff83374e24a2525daea30da3a6 P 29a8cd4ab84b49e49daa7ec296c646d7: Deleting consensus metadata
I20260810 07:24:59.676517 1038 catalog_manager.cc:4841] TS 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709): tablet f28367ff83374e24a2525daea30da3a6 (table test-table [id=5271ca15368b4c91a1c034cec72557c2]) successfully deleted
I20260810 07:24:59.676510 1058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 07:24:59.676674 1038 catalog_manager.cc:4841] TS 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077): tablet f28367ff83374e24a2525daea30da3a6 (table test-table [id=5271ca15368b4c91a1c034cec72557c2]) successfully deleted
W20260810 07:24:59.676699 1058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:24:59.677026 1723 ts_tablet_manager.cc:1905] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:24:59.678036 1215 tablet_service.cc:1467] Processing CreateTablet for tablet 91863626667849be989f8d31f8c8eece (DEFAULT_TABLE table=test-table [id=4f486c49df7f4d00a60169d5e98f64b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:59.678071 1373 tablet_service.cc:1467] Processing CreateTablet for tablet 91863626667849be989f8d31f8c8eece (DEFAULT_TABLE table=test-table [id=4f486c49df7f4d00a60169d5e98f64b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:59.678174 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91863626667849be989f8d31f8c8eece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.678184 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91863626667849be989f8d31f8c8eece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.678552 1723 ts_tablet_manager.cc:1918] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:24:59.678606 1723 log.cc:1196] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/f28367ff83374e24a2525daea30da3a6
I20260810 07:24:59.678821 1723 ts_tablet_manager.cc:1939] T f28367ff83374e24a2525daea30da3a6 P 8b0541e5ccc74b63907465503853adb3: Deleting consensus metadata
I20260810 07:24:59.679471 1647 tablet_bootstrap.cc:492] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Bootstrap starting.
I20260810 07:24:59.679483 1645 tablet_bootstrap.cc:492] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7: Bootstrap starting.
I20260810 07:24:59.679774 1647 tablet_bootstrap.cc:654] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.679816 1645 tablet_bootstrap.cc:654] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.680197 1040 catalog_manager.cc:4841] TS 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951): tablet f28367ff83374e24a2525daea30da3a6 (table test-table [id=5271ca15368b4c91a1c034cec72557c2]) successfully deleted
I20260810 07:24:59.680292 1537 tablet_service.cc:1467] Processing CreateTablet for tablet 91863626667849be989f8d31f8c8eece (DEFAULT_TABLE table=test-table [id=4f486c49df7f4d00a60169d5e98f64b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:24:59.680397 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91863626667849be989f8d31f8c8eece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.681309 1646 tablet_bootstrap.cc:492] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Bootstrap starting.
I20260810 07:24:59.681484 1645 tablet_bootstrap.cc:492] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7: No bootstrap required, opened a new log
I20260810 07:24:59.681528 1645 ts_tablet_manager.cc:1397] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:24:59.681648 1645 raft_consensus.cc:348] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.681715 1645 raft_consensus.cc:374] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.681669 1646 tablet_bootstrap.cc:654] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:59.681738 1645 raft_consensus.cc:729] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a8cd4ab84b49e49daa7ec296c646d7, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.681783 1645 consensus_queue.cc:260] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.681865 1645 ts_tablet_manager.cc:1428] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.682446 1646 tablet_bootstrap.cc:492] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: No bootstrap required, opened a new log
I20260810 07:24:59.682505 1646 ts_tablet_manager.cc:1397] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 07:24:59.682528 1647 tablet_bootstrap.cc:492] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: No bootstrap required, opened a new log
I20260810 07:24:59.682574 1647 ts_tablet_manager.cc:1397] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:24:59.682627 1646 raft_consensus.cc:348] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.682704 1646 raft_consensus.cc:374] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.682725 1646 raft_consensus.cc:729] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.682721 1647 raft_consensus.cc:348] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.682787 1647 raft_consensus.cc:374] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.682814 1647 raft_consensus.cc:729] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0541e5ccc74b63907465503853adb3, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.682821 1646 consensus_queue.cc:260] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.682857 1647 consensus_queue.cc:260] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.682938 1646 ts_tablet_manager.cc:1428] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.682948 1647 ts_tablet_manager.cc:1428] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:24:59.937458 1657 raft_consensus.cc:482] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:24:59.937623 1657 raft_consensus.cc:504] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.937928 1657 leader_election.cc:288] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077), 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951)
I20260810 07:24:59.938156 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91863626667849be989f8d31f8c8eece" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" is_pre_election: true
I20260810 07:24:59.938190 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91863626667849be989f8d31f8c8eece" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3" is_pre_election: true
I20260810 07:24:59.938280 1235 raft_consensus.cc:2454] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 0.
I20260810 07:24:59.938319 1397 raft_consensus.cc:2454] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 0.
I20260810 07:24:59.938479 1480 leader_election.cc:302] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 60ee73b1103345368e5ad9ba920ecab6; no voters:
I20260810 07:24:59.938642 1657 raft_consensus.cc:2781] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:24:59.938690 1657 raft_consensus.cc:482] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:24:59.938705 1657 raft_consensus.cc:3037] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.939465 1657 raft_consensus.cc:504] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.939739 1657 leader_election.cc:288] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 election: Requested vote from peers 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077), 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951)
I20260810 07:24:59.940002 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91863626667849be989f8d31f8c8eece" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a8cd4ab84b49e49daa7ec296c646d7"
I20260810 07:24:59.940056 1397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91863626667849be989f8d31f8c8eece" candidate_uuid: "60ee73b1103345368e5ad9ba920ecab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0541e5ccc74b63907465503853adb3"
I20260810 07:24:59.940111 1235 raft_consensus.cc:3037] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.940161 1397 raft_consensus.cc:3037] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.940842 1235 raft_consensus.cc:2454] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 1.
I20260810 07:24:59.940960 1397 raft_consensus.cc:2454] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ee73b1103345368e5ad9ba920ecab6 in term 1.
I20260810 07:24:59.941025 1480 leader_election.cc:302] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a8cd4ab84b49e49daa7ec296c646d7, 60ee73b1103345368e5ad9ba920ecab6; no voters:
I20260810 07:24:59.941167 1657 raft_consensus.cc:2781] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:59.941324 1657 raft_consensus.cc:686] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 1 LEADER]: Becoming Leader. State: Replica: 60ee73b1103345368e5ad9ba920ecab6, State: Running, Role: LEADER
I20260810 07:24:59.941411 1657 consensus_queue.cc:237] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } }
I20260810 07:24:59.942071 1058 catalog_manager.cc:5495] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195). New cstate: current_term: 1 leader_uuid: "60ee73b1103345368e5ad9ba920ecab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ee73b1103345368e5ad9ba920ecab6" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36709 } health_report { overall_health: HEALTHY } } }
I20260810 07:24:59.971889 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.969097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.24.60.195:36709)
0810 07:24:59.969352 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.969354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.969363 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:24:59.969384 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.969385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.969396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.969465 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.969616 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.970172 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.971062 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.971288 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.971355 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.971561 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.971596 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.971659 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.971782 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.971808 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":177,"threads_started":1}
I20260810 07:24:59.977128 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.974936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:42399 (local address 127.24.60.193:45077)
0810 07:24:59.974989 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.974995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.975051 (+ 56us) server_negotiation.cc:407] Connection header received
0810 07:24:59.975109 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.975110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.975120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.975166 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.975306 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.975759 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.976536 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.976722 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.976780 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.976930 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.976957 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.976997 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.977052 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.977068 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:24:59.977218 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.974741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.24.60.195:42399)
0810 07:24:59.974986 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.975004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.975066 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.975192 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.975194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.975195 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:59.975239 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.975240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.975850 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.975851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.976422 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.976423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.976795 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.976796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.976840 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.976856 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.977013 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.977014 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.977051 (+ 37us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.977090 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260810 07:24:59.979482 1397 raft_consensus.cc:1261] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Refusing update from remote peer 60ee73b1103345368e5ad9ba920ecab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.979493 1235 raft_consensus.cc:1261] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Refusing update from remote peer 60ee73b1103345368e5ad9ba920ecab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:24:59.979789 1754 consensus_queue.cc:1035] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0541e5ccc74b63907465503853adb3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.979990 1657 consensus_queue.cc:1035] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a8cd4ab84b49e49daa7ec296c646d7" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:24:59.987777 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.985244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:52417 (local address 127.24.60.195:36709)
0810 07:24:59.985291 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.985293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.985426 (+ 133us) server_negotiation.cc:407] Connection header received
0810 07:24:59.985508 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.985509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.985520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.985577 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:24:59.985666 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.986132 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.987021 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.987298 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.987390 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.987532 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.987570 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.987619 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.987676 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.987701 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:24:59.987815 1764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.985217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36709 (local address 127.24.60.193:52417)
0810 07:24:59.985361 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.985376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.985443 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.985578 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.985579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.985581 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:24:59.985626 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.985627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.986215 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.986216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.986854 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.986857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.987383 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.987384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.987450 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.987456 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:24:59.987621 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:24:59.987624 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.987673 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.987731 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260810 07:24:59.997365 1058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:59.997485 1058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:24:59.998375 1058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc73832166f4f6288094fd3ad6f5a97: Sending DeleteTablet for 3 replicas of tablet 91863626667849be989f8d31f8c8eece
I20260810 07:24:59.998692 1215 tablet_service.cc:1514] Processing DeleteTablet for tablet 91863626667849be989f8d31f8c8eece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:46032
I20260810 07:24:59.998797 1373 tablet_service.cc:1514] Processing DeleteTablet for tablet 91863626667849be989f8d31f8c8eece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49448
I20260810 07:24:59.998824 1537 tablet_service.cc:1514] Processing DeleteTablet for tablet 91863626667849be989f8d31f8c8eece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:24:59 UTC) from {username='oryx', principal='oryx/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:44130
I20260810 07:24:59.998878 1724 tablet_replica.cc:330] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7: stopping tablet replica
I20260810 07:24:59.998916 1725 tablet_replica.cc:330] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: stopping tablet replica
I20260810 07:24:59.998942 1724 raft_consensus.cc:2229] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:59.998981 1725 raft_consensus.cc:2229] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:24:59.998986 1724 raft_consensus.cc:2258] T 91863626667849be989f8d31f8c8eece P 29a8cd4ab84b49e49daa7ec296c646d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.999055 1725 raft_consensus.cc:2258] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:24:59.999069 1723 tablet_replica.cc:330] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: stopping tablet replica
I20260810 07:24:59.999076 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 1139
I20260810 07:24:59.999130 1723 raft_consensus.cc:2229] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:24:59.999172 1723 raft_consensus.cc:2258] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:00.004091 1723 ts_tablet_manager.cc:1905] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:00.004324 1725 ts_tablet_manager.cc:1905] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:00.006359 1725 ts_tablet_manager.cc:1918] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:00.006462 1723 ts_tablet_manager.cc:1918] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:00.006496 1725 log.cc:1196] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/91863626667849be989f8d31f8c8eece
I20260810 07:25:00.006573 1723 log.cc:1196] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/91863626667849be989f8d31f8c8eece
I20260810 07:25:00.006779 1725 ts_tablet_manager.cc:1939] T 91863626667849be989f8d31f8c8eece P 60ee73b1103345368e5ad9ba920ecab6: Deleting consensus metadata
I20260810 07:25:00.006826 1723 ts_tablet_manager.cc:1939] T 91863626667849be989f8d31f8c8eece P 8b0541e5ccc74b63907465503853adb3: Deleting consensus metadata
I20260810 07:25:00.007321 1038 catalog_manager.cc:4841] TS 60ee73b1103345368e5ad9ba920ecab6 (127.24.60.195:36709): tablet 91863626667849be989f8d31f8c8eece (table test-table [id=4f486c49df7f4d00a60169d5e98f64b6]) successfully deleted
I20260810 07:25:00.007390 1040 catalog_manager.cc:4841] TS 8b0541e5ccc74b63907465503853adb3 (127.24.60.194:45951): tablet 91863626667849be989f8d31f8c8eece (table test-table [id=4f486c49df7f4d00a60169d5e98f64b6]) successfully deleted
W20260810 07:25:00.012691 1038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:45077 (error 108)
I20260810 07:25:00.013031 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 1300
I20260810 07:25:00.018208 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.017954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:45077 (local address 127.0.0.1:46056)
0810 07:25:00.018100 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.018160 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
W20260810 07:25:00.018416 1038 catalog_manager.cc:4584] TS 29a8cd4ab84b49e49daa7ec296c646d7 (127.24.60.193:45077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91863626667849be989f8d31f8c8eece: Network error: Client connection negotiation failed: client connection to 127.24.60.193:45077: connect: Connection refused (error 111)
I20260810 07:25:00.023384 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 1462
I20260810 07:25:00.033034 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 1018
2026-08-10T07:25:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3456 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:00 dist-test-slave-vxj2 krb5kdc[1770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:00 dist-test-slave-vxj2 krb5kdc[1770](info): set up 2 sockets
Aug 10 07:25:00 dist-test-slave-vxj2 krb5kdc[1770](info): commencing operation
krb5kdc: starting...
W20260810 07:25:01.271543 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:25:01 dist-test-slave-vxj2 krb5kdc[1770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:25:01.409628 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35781
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:01.496358 1785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:01.496594 1785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:01.496619 1785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:01.497810 1785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:01.497864 1785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:01.497879 1785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:01.497892 1785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:01.497905 1785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:01.499490 1785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:01.499734 1785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:01.499935 1785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:01.505244 1790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:01.505262 1793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:01.505244 1791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:01.505630 1785 server_base.cc:1034] running on GCE node
I20260810 07:25:01.505909 1785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:01.506233 1785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:01.507402 1785 hybrid_clock.cc:648] HybridClock initialized: now 1786346701507378 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T07:25:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1480 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:01 dist-test-slave-vxj2 krb5kdc[1798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:01 dist-test-slave-vxj2 krb5kdc[1798](info): set up 2 sockets
Aug 10 07:25:01 dist-test-slave-vxj2 krb5kdc[1798](info): commencing operation
krb5kdc: starting...
W20260810 07:25:02.651394 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:25:02 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:25:02.781345 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42049
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:02.866484 1815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:02.866729 1815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:02.866755 1815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:02.868031 1815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:02.868083 1815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:02.868096 1815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:02.868109 1815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:02.868153 1815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:02.869745 1815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:02.870028 1815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:02.870275 1815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:02.875483 1822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.875604 1815 server_base.cc:1034] running on GCE node
W20260810 07:25:02.875546 1823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.875548 1826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.876044 1815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:02.876389 1815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:02.877609 1815 hybrid_clock.cc:648] HybridClock initialized: now 1786346702877587 us; error 45 us; skew 500 ppm
Aug 10 07:25:02 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:02.880441 1815 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:25:02.880939 1815 webserver.cc:460] Webserver started at http://127.24.60.254:40867/ using document root <none> and password file <none>
I20260810 07:25:02.881253 1815 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:02.881305 1815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:02.881488 1815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:02.882701 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "200e1506260e49faa68584be20581389"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-vxj2"
server_key: "7a1f7e0bbc0268135b8b347dc515c3b3"
server_key_iv: "3941c79677af992162d33e514bb7b1c9"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.883085 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "200e1506260e49faa68584be20581389"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-vxj2"
server_key: "7a1f7e0bbc0268135b8b347dc515c3b3"
server_key_iv: "3941c79677af992162d33e514bb7b1c9"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.884589 1815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:25:02.885416 1835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.885622 1815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:02.885720 1815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "200e1506260e49faa68584be20581389"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-vxj2"
server_key: "7a1f7e0bbc0268135b8b347dc515c3b3"
server_key_iv: "3941c79677af992162d33e514bb7b1c9"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.885797 1815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:02.918264 1815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:02.920787 1815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:02.920960 1815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:02.925050 1815 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:42049
I20260810 07:25:02.925107 1894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:42049 every 8 connection(s)
I20260810 07:25:02.925522 1815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:02.926450 1895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:02.927532 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 1815
I20260810 07:25:02.927664 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:25:02.927886 24819 external_mini_cluster.cc:1428] Setting key 503554219628423971a11e57ef3fe999
I20260810 07:25:02.929589 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Bootstrap starting.
I20260810 07:25:02.930418 1895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:02.930734 1895 log.cc:824] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:02.931515 1895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: No bootstrap required, opened a new log
I20260810 07:25:02.934188 1895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:02.934405 1895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:02.934448 1895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200e1506260e49faa68584be20581389, State: Initialized, Role: FOLLOWER
I20260810 07:25:02.934621 1895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:02.934707 1895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:02.934741 1895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:02.934788 1895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 07:25:02 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:02.935444 1895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:02.935567 1895 leader_election.cc:302] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200e1506260e49faa68584be20581389; no voters:
I20260810 07:25:02.935814 1895 leader_election.cc:288] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:02.935842 1901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:02.935971 1901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 LEADER]: Becoming Leader. State: Replica: 200e1506260e49faa68584be20581389, State: Running, Role: LEADER
I20260810 07:25:02.936102 1901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:02.936122 1895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:02.936534 1903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "200e1506260e49faa68584be20581389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } } }
I20260810 07:25:02.936669 1903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:02.936534 1902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200e1506260e49faa68584be20581389. Latest consensus state: current_term: 1 leader_uuid: "200e1506260e49faa68584be20581389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } } }
I20260810 07:25:02.936815 1902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:02.937233 1907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:02.937760 1907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:02.939550 1907 catalog_manager.cc:1347] Generated new cluster ID: b592bace2fc04c08bbd8d2c87a45f26d
I20260810 07:25:02.939604 1907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:02.941124 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.928475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.24.60.254:42049)
0810 07:25:02.928661 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.928669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.928695 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:25:02.929299 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.929313 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.929375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.929506 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.930383 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.931568 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.932668 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.933092 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.933238 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.936073 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.936082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.936083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.936097 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.939281 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.939765 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.939769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.939769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.939886 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.940149 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.940150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.940150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.940448 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.940588 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.940770 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.940910 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260810 07:25:02.944710 1907 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:02.945318 1907 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:02.956243 1907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Generated new TSK 0
I20260810 07:25:02.956514 1907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:25:02.988718 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42049
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:03.087725 1926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:03.087950 1926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:03.087976 1926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:03.089334 1926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:03.089380 1926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:03.089408 1926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:25:03.090941 1926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:03.091198 1926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:03.091462 1926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:03.096840 1934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:03.096903 1926 server_base.cc:1034] running on GCE node
W20260810 07:25:03.096849 1936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.096918 1932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:03.097359 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:03.097591 1926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:03.098769 1926 hybrid_clock.cc:648] HybridClock initialized: now 1786346703098740 us; error 42 us; skew 500 ppm
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:03.101476 1926 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:25:03.102003 1926 webserver.cc:460] Webserver started at http://127.24.60.193:37971/ using document root <none> and password file <none>
I20260810 07:25:03.102247 1926 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:03.102319 1926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:03.102481 1926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:03.103637 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "6a19c72470a94e299b88787ce046d3b7"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0d74ab04bd3bad491ff6eb43df1af0d9"
server_key_iv: "d2203df4922dc05a40d1b2c542556ea4"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.104046 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "6a19c72470a94e299b88787ce046d3b7"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0d74ab04bd3bad491ff6eb43df1af0d9"
server_key_iv: "d2203df4922dc05a40d1b2c542556ea4"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.105538 1926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:25:03.106400 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.106598 1926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:03.106710 1926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "6a19c72470a94e299b88787ce046d3b7"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0d74ab04bd3bad491ff6eb43df1af0d9"
server_key_iv: "d2203df4922dc05a40d1b2c542556ea4"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.106788 1926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:03.127650 1926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:03.129763 1926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:03.129956 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:03.130236 1926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:03.130574 1926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:03.130631 1926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.130674 1926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:03.130719 1926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.137045 1926 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36187
I20260810 07:25:03.137104 2073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36187 every 8 connection(s)
I20260810 07:25:03.137557 1926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:03.145572 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 1926
I20260810 07:25:03.145720 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:03.146011 24819 external_mini_cluster.cc:1428] Setting key 275e812e9711876335dcc169f530daf3
I20260810 07:25:03.148310 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.138508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:52705 (local address 127.24.60.254:42049)
0810 07:25:03.138555 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.138557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.138780 (+ 223us) server_negotiation.cc:407] Connection header received
0810 07:25:03.139354 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.139356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.139368 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.139453 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.140502 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.141096 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.142213 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.142439 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.142512 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.144872 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:03.144874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:03.144875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:03.144884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:03.146663 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.147175 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.147177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.147177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.147248 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.147546 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.147547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.147548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.147680 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:03.147768 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.148101 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.148205 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:25:03.148527 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.138462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:52705)
0810 07:25:03.138635 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.138680 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.139198 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.139529 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.139532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.140170 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.140383 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.140389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.141240 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.141241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.142103 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.142105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.142527 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.142528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.142630 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.143184 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.143193 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.144752 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:03.146768 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.146778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.146781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.147067 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.147330 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.147331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.147332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.147465 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.147758 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:03.147767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:03.147936 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.148063 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.148257 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260810 07:25:03.149071 2074 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42049
I20260810 07:25:03.149237 2074 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:03.149427 2074 heartbeater.cc:511] Master 127.24.60.254:42049 requested a full tablet report, sending...
I20260810 07:25:03.150112 1853 ts_manager.cc:194] Registered new tserver with Master: 6a19c72470a94e299b88787ce046d3b7 (127.24.60.193:36187)
I20260810 07:25:03.151062 1853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:52705
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:25:03.176960 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42049
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:03.266667 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:03.266949 2079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:03.266992 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:03.268277 2079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:03.268360 2079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:03.268407 2079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:25:03.270053 2079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:03.270366 2079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:03.270615 2079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:03.276041 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.276041 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.276067 2085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:03.276418 2079 server_base.cc:1034] running on GCE node
I20260810 07:25:03.276700 2079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:03.276989 2079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:03.278196 2079 hybrid_clock.cc:648] HybridClock initialized: now 1786346703278166 us; error 46 us; skew 500 ppm
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:03.280836 2079 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:25:03.281333 2079 webserver.cc:460] Webserver started at http://127.24.60.194:46749/ using document root <none> and password file <none>
I20260810 07:25:03.281565 2079 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:03.281610 2079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:03.281708 2079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:03.282805 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "dfb34f752e2a4d87a1f890b462772156"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0a9f9ebb2f105bc92668f2812c5f3d08"
server_key_iv: "4e310216d6c2929ef74a0bebefbecdeb"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.283116 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "dfb34f752e2a4d87a1f890b462772156"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0a9f9ebb2f105bc92668f2812c5f3d08"
server_key_iv: "4e310216d6c2929ef74a0bebefbecdeb"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.284515 2079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:25:03.285329 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.285509 2079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:03.285606 2079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "dfb34f752e2a4d87a1f890b462772156"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0a9f9ebb2f105bc92668f2812c5f3d08"
server_key_iv: "4e310216d6c2929ef74a0bebefbecdeb"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.285670 2079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:03.306066 2079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:03.308646 2079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:03.308776 2079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:03.308986 2079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:03.309414 2079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:03.309454 2079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.309478 2079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:03.309516 2079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.315881 2079 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:46507
I20260810 07:25:03.316068 2227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:46507 every 8 connection(s)
I20260810 07:25:03.316264 2079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:03.323036 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 2079
I20260810 07:25:03.323168 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:03.323402 24819 external_mini_cluster.cc:1428] Setting key 20b5b491053a71e30c42d8ab06751722
I20260810 07:25:03.326637 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.317108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:35957 (local address 127.24.60.254:42049)
0810 07:25:03.317238 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.317240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.317381 (+ 141us) server_negotiation.cc:407] Connection header received
0810 07:25:03.317892 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.317893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.317906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.317981 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.319177 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.319711 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.320769 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.320986 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.321046 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.323332 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:03.323334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:03.323334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:03.323342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:03.324886 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.325435 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.325437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.325437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.325510 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.325870 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.325872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.325872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.326023 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:03.326084 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.326470 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.326566 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 07:25:03.326953 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.317055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:35957)
0810 07:25:03.317240 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.317285 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.317769 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.318042 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.318046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.318654 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.319068 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.319077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.319830 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.319832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.320683 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.320685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.321067 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.321068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.321198 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.321712 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.321720 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.323227 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:03.324999 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.325009 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.325012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.325353 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.325590 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.325591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.325591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.325756 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.326097 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:03.326105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:03.326303 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.326423 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.326649 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":117,"threads_started":1}
I20260810 07:25:03.327522 2228 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42049
I20260810 07:25:03.327638 2228 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:03.327852 2228 heartbeater.cc:511] Master 127.24.60.254:42049 requested a full tablet report, sending...
I20260810 07:25:03.328241 1853 ts_manager.cc:194] Registered new tserver with Master: dfb34f752e2a4d87a1f890b462772156 (127.24.60.194:46507)
I20260810 07:25:03.328719 1853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:35957
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:25:03.353284 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42049
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:03.441568 2233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:03.441808 2233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:03.441835 2233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:03.443068 2233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:03.443120 2233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:03.443146 2233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:25:03.444661 2233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:03.444937 2233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:03.445235 2233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:03.450512 2241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.450577 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.450589 2244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:03.450821 2233 server_base.cc:1034] running on GCE node
I20260810 07:25:03.451095 2233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:03.451350 2233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:03.452522 2233 hybrid_clock.cc:648] HybridClock initialized: now 1786346703452508 us; error 28 us; skew 500 ppm
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:03.455241 2233 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:25:03.455686 2233 webserver.cc:460] Webserver started at http://127.24.60.195:45929/ using document root <none> and password file <none>
I20260810 07:25:03.455901 2233 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:03.456003 2233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:03.456182 2233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:03.457397 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "8448d19b84564f799882ebd0b2d4d9d9"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "1ea6f18fc2156f3d4ab592f95f2e951f"
server_key_iv: "f67b2ce74e7db81a33a1911f11341ef7"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.457785 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "8448d19b84564f799882ebd0b2d4d9d9"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "1ea6f18fc2156f3d4ab592f95f2e951f"
server_key_iv: "f67b2ce74e7db81a33a1911f11341ef7"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.459359 2233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:03.460493 2251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.460878 2233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:03.461025 2233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "8448d19b84564f799882ebd0b2d4d9d9"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "1ea6f18fc2156f3d4ab592f95f2e951f"
server_key_iv: "f67b2ce74e7db81a33a1911f11341ef7"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.461118 2233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:03.480280 2233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:03.482882 2233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:03.483062 2233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:03.483362 2233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:03.483721 2233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:03.483755 2233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.483779 2233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:03.483829 2233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.490697 2233 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:46233
I20260810 07:25:03.490825 2381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:46233 every 8 connection(s)
I20260810 07:25:03.491158 2233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:03.499511 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 2233
I20260810 07:25:03.499646 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:03.499909 24819 external_mini_cluster.cc:1428] Setting key 348cdba5e83f4517609fb8d37504bf35
I20260810 07:25:03.501729 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.492001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:49093 (local address 127.24.60.254:42049)
0810 07:25:03.492094 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.492097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.492303 (+ 206us) server_negotiation.cc:407] Connection header received
0810 07:25:03.492807 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.492808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.492820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.492879 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.494022 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.494533 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.495569 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.495760 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.495813 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.498165 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:03.498167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:03.498167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:03.498175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:03.499799 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.500437 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.500438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.500438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.500519 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.500936 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.500937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.500937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.501094 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:03.501193 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.501563 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.501658 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:25:03.501850 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.491951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:49093)
0810 07:25:03.492159 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.492200 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.492706 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.492943 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.492947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.493672 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.493889 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.493895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.494640 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.494641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.495487 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.495489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.495835 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.495836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.495956 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.496495 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.496504 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.498068 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:03.499888 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.499905 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.499908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.500310 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.500588 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.500590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.500590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.500848 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.501228 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:03.501236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:03.501400 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.501502 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.501643 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260810 07:25:03.502381 2383 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42049
I20260810 07:25:03.502499 2383 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:03.502729 2383 heartbeater.cc:511] Master 127.24.60.254:42049 requested a full tablet report, sending...
I20260810 07:25:03.503155 1853 ts_manager.cc:194] Registered new tserver with Master: 8448d19b84564f799882ebd0b2d4d9d9 (127.24.60.195:46233)
I20260810 07:25:03.503551 1853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:49093
I20260810 07:25:03.512589 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab.
Entry for principal oryx/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab.
WARNING: no policy specified for oryx/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab.
Entry for principal oryx/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab.
WARNING: no policy specified for oryx/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab.
Entry for principal oryx/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab.
I20260810 07:25:03.596661 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 1926
I20260810 07:25:03.605706 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 2079
I20260810 07:25:03.619342 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 2233
I20260810 07:25:03.625221 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 1815
I20260810 07:25:03.633404 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42049
--webserver_interface=127.24.60.254
--webserver_port=40867
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:03.718564 2394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:03.718830 2394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:03.718876 2394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:03.720104 2394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:03.720177 2394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:03.720202 2394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:03.720278 2394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:03.720312 2394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:03.721925 2394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:03.722208 2394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:03.722456 2394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:03.727451 2400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.727428 2403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.727425 2401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:03.728493 2394 server_base.cc:1034] running on GCE node
I20260810 07:25:03.728740 2394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:03.729007 2394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:03.730154 2394 hybrid_clock.cc:648] HybridClock initialized: now 1786346703730140 us; error 36 us; skew 500 ppm
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:03.732656 2394 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:25:03.733201 2394 webserver.cc:460] Webserver started at http://127.24.60.254:40867/ using document root <none> and password file <none>
I20260810 07:25:03.733474 2394 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:03.733568 2394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:03.734995 2394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:03.735661 2411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.735836 2394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:25:03.735930 2394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "200e1506260e49faa68584be20581389"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-vxj2"
server_key: "7a1f7e0bbc0268135b8b347dc515c3b3"
server_key_iv: "3941c79677af992162d33e514bb7b1c9"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.736338 2394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:03.761667 2394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:03.763726 2394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:03.763907 2394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:03.768085 2394 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:42049
I20260810 07:25:03.768144 2470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:42049 every 8 connection(s)
I20260810 07:25:03.768573 2394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:03.768992 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 2394
I20260810 07:25:03.769342 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:36187
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:42049
--webserver_port=37971
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:25:03.770401 2471 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:25:03.771471 2471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Bootstrap starting.
I20260810 07:25:03.780371 2471 log.cc:824] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:03.782299 2471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:25:03.782648 2471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Bootstrap complete.
I20260810 07:25:03.785007 2471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:03.785307 2471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200e1506260e49faa68584be20581389, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.785493 2471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:03.785565 2471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:03.785629 2471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:03.785677 2471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:25:03.786780 2471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:03.786875 2471 leader_election.cc:302] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200e1506260e49faa68584be20581389; no voters:
I20260810 07:25:03.787115 2471 leader_election.cc:288] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:25:03.787153 2475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:25:03.787288 2475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [term 2 LEADER]: Becoming Leader. State: Replica: 200e1506260e49faa68584be20581389, State: Running, Role: LEADER
I20260810 07:25:03.787376 2475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } }
I20260810 07:25:03.787407 2471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:03.788856 2477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "200e1506260e49faa68584be20581389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } } }
I20260810 07:25:03.788949 2477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:03.789209 2476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200e1506260e49faa68584be20581389. Latest consensus state: current_term: 2 leader_uuid: "200e1506260e49faa68584be20581389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e1506260e49faa68584be20581389" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42049 } } }
I20260810 07:25:03.789270 2476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:03.789343 2491 catalog_manager.cc:1259] Loaded cluster ID: b592bace2fc04c08bbd8d2c87a45f26d
I20260810 07:25:03.789577 2491 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: loading cluster ID for follower catalog manager: success
I20260810 07:25:03.790772 2491 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: acquiring CA information for follower catalog manager: success
I20260810 07:25:03.791112 2491 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:25:03.791203 2494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:03.791400 2494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:03.791482 2494 catalog_manager.cc:1259] Loaded cluster ID: b592bace2fc04c08bbd8d2c87a45f26d
I20260810 07:25:03.791513 2494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:03.791679 2494 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:03.791772 2494 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 200e1506260e49faa68584be20581389: Loaded TSK: 0
I20260810 07:25:03.791903 2494 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:25:03.856917 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:03.857193 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:03.857254 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:03.858518 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:03.858592 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:03.858640 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:25:03.860142 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.193.keytab
--principal=oryx/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:36187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=37971
--tserver_master_addrs=127.24.60.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:03.860414 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:03.860652 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:03.866961 2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.866952 2502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:03.866946 2504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:03.867203 2473 server_base.cc:1034] running on GCE node
I20260810 07:25:03.867620 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:03.867921 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:03.869087 2473 hybrid_clock.cc:648] HybridClock initialized: now 1786346703869071 us; error 35 us; skew 500 ppm
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346703, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:03.872247 2473 init.cc:377] Logged in from keytab as oryx/127.24.60.193@KRBTEST.COM (short username oryx)
I20260810 07:25:03.872833 2473 webserver.cc:460] Webserver started at http://127.24.60.193:37971/ using document root <none> and password file <none>
I20260810 07:25:03.873116 2473 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:03.873242 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:03.874893 2473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:03.875667 2515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.875905 2473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:03.876024 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "6a19c72470a94e299b88787ce046d3b7"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0d74ab04bd3bad491ff6eb43df1af0d9"
server_key_iv: "d2203df4922dc05a40d1b2c542556ea4"
server_key_version: "encryptionkey@0"
I20260810 07:25:03.876432 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:03.908015 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:03.910645 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:03.910915 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:03.911233 2473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:03.911626 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:03.911690 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.911731 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:03.911783 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:03.918689 2473 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36187
I20260810 07:25:03.918748 2644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36187 every 8 connection(s)
I20260810 07:25:03.919170 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:03.925683 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 2473
I20260810 07:25:03.926045 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:46507
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:42049
--webserver_port=46749
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:03.928251 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.920025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:39229)
0810 07:25:03.920208 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.920262 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.920893 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.921368 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.921371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.921957 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.922190 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.922195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.923261 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.923261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.924057 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.924059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.924622 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.924623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.924723 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.925224 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.925232 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.928011 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
W20260810 07:25:03.928277 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:03.920154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:39229 (local address 127.24.60.254:42049)
0810 07:25:03.920355 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.920365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.920411 (+ 46us) server_negotiation.cc:407] Connection header received
0810 07:25:03.921116 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.921124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.921206 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.921314 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.922296 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.923169 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.924135 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.924537 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.924619 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.928148 (+ 3529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:39229: BlockingRecv error: recv got EOF from 127.24.60.193:39229 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
W20260810 07:25:03.928555 2645 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:42049 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.60.254:42049: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:03.933744 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.928824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:55495)
0810 07:25:03.928873 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.928886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.928954 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.929106 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.929107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.929306 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.929365 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.929368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.929999 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.930000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.930888 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.930892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.931260 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.931261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.931340 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.931763 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.931769 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.933694 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 07:25:03.933923 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:03.928867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:55495 (local address 127.24.60.254:42049)
0810 07:25:03.928929 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.928931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.928938 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:25:03.929017 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.929018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.929030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.929078 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.929444 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.929918 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.930993 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.931184 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.931236 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.933835 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:55495: BlockingRecv error: recv got EOF from 127.24.60.193:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:03 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:03.938665 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.934028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:35595)
0810 07:25:03.934218 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.934228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.934281 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.934419 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.934419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.934540 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.934566 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.934567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.935511 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.935511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.935975 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.935976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.936471 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.936472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.936508 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.936959 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.936964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.938616 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":138,"threads_started":1}
W20260810 07:25:03.938786 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:03.934076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:35595 (local address 127.24.60.254:42049)
0810 07:25:03.934134 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.934136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.934266 (+ 130us) server_negotiation.cc:407] Connection header received
0810 07:25:03.934327 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.934328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.934341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.934410 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.934638 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.935431 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.936055 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.936381 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.936455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.938724 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:35595: BlockingRecv error: recv got EOF from 127.24.60.193:35595 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 07:25:03.938819 2645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:25:04.012717 2649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:04.012952 2649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:04.012979 2649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:04.014262 2649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:04.014323 2649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:04.014353 2649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:25:04.015957 2649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.194.keytab
--principal=oryx/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:46507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=46749
--tserver_master_addrs=127.24.60.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:04.016201 2649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:04.016392 2649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:04.021715 2660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:04.022011 2658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:04.022197 2649 server_base.cc:1034] running on GCE node
W20260810 07:25:04.022228 2656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:04.022555 2649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:04.022776 2649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:04.023921 2649 hybrid_clock.cc:648] HybridClock initialized: now 1786346704023910 us; error 31 us; skew 500 ppm
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346704, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:04.026562 2649 init.cc:377] Logged in from keytab as oryx/127.24.60.194@KRBTEST.COM (short username oryx)
I20260810 07:25:04.027069 2649 webserver.cc:460] Webserver started at http://127.24.60.194:46749/ using document root <none> and password file <none>
I20260810 07:25:04.027302 2649 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:04.027382 2649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:04.028771 2649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:04.029426 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:04.029610 2649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:04.029706 2649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "dfb34f752e2a4d87a1f890b462772156"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "0a9f9ebb2f105bc92668f2812c5f3d08"
server_key_iv: "4e310216d6c2929ef74a0bebefbecdeb"
server_key_version: "encryptionkey@0"
I20260810 07:25:04.030074 2649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:04.051028 2649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:04.053057 2649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:04.053247 2649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:04.053542 2649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:04.053894 2649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:04.053956 2649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:04.053992 2649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:04.054020 2649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:04.059904 2649 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:46507
I20260810 07:25:04.059970 2797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:46507 every 8 connection(s)
I20260810 07:25:04.060325 2649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:04.062561 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 2649
I20260810 07:25:04.062856 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:46233
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:42049
--webserver_port=45929
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:04.068387 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.061150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:55799 (local address 127.24.60.254:42049)
0810 07:25:04.061225 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.061227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.061473 (+ 246us) server_negotiation.cc:407] Connection header received
0810 07:25:04.062020 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.062021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.062039 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.062105 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.063245 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.063753 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.064789 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.065013 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.065106 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.068326 (+ 3220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:55799: BlockingRecv error: recv got EOF from 127.24.60.194:55799 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:25:04.068464 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.061143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:55799)
0810 07:25:04.061338 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.061378 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.061903 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.062184 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.062188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.062846 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.063134 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.063147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.063864 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.063865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.064686 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.064688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.065121 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.065122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.065310 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.065881 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.065890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.068232 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260810 07:25:04.068693 2798 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:42049 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.60.254:42049: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:04.073457 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.068951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:39381)
0810 07:25:04.069005 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.069017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.069059 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.069239 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.069240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.069385 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.069436 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.069438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.070099 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.070100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.070616 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.070617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.070984 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.070985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.071058 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.071567 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.071571 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.073393 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 07:25:04.073609 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.069031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:39381 (local address 127.24.60.254:42049)
0810 07:25:04.069085 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.069087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.069097 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:25:04.069125 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.069126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.069136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.069234 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.069525 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.070017 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.070703 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.070903 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.070986 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.073538 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:39381: BlockingRecv error: recv got EOF from 127.24.60.194:39381 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:04.078406 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.073788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:36553)
0810 07:25:04.073831 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.073842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.073878 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.074051 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.074051 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.074160 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.074193 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.074194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.075126 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.075127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.075607 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.075609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.076070 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.076070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.076108 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.076543 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.076548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.078330 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 07:25:04.078491 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.073875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:36553 (local address 127.24.60.254:42049)
0810 07:25:04.073916 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.073918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.073927 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:04.073950 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.073950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.073959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.074039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.074262 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.075049 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.075684 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.075998 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.076066 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.078414 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:36553: BlockingRecv error: recv got EOF from 127.24.60.194:36553 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 07:25:04.078552 2798 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:25:04.151022 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:04.151250 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:04.151278 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:04.152655 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:04.152719 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:04.152777 2804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:25:04.154417 2804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/krb5kdc/oryx_127.24.60.195.keytab
--principal=oryx/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:46233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=45929
--tserver_master_addrs=127.24.60.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:04.154695 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:04.154942 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:04.160288 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:04.160312 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:04.160281 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:04.160751 2804 server_base.cc:1034] running on GCE node
I20260810 07:25:04.161008 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:04.161316 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:04.162485 2804 hybrid_clock.cc:648] HybridClock initialized: now 1786346704162471 us; error 30 us; skew 500 ppm
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346704, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:04.165279 2804 init.cc:377] Logged in from keytab as oryx/127.24.60.195@KRBTEST.COM (short username oryx)
I20260810 07:25:04.165768 2804 webserver.cc:460] Webserver started at http://127.24.60.195:45929/ using document root <none> and password file <none>
I20260810 07:25:04.165984 2804 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:04.166043 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:04.167539 2804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:04.168211 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:04.168431 2804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:04.168511 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "8448d19b84564f799882ebd0b2d4d9d9"
format_stamp: "Formatted at 2026-08-10 07:25:03 on dist-test-slave-vxj2"
server_key: "1ea6f18fc2156f3d4ab592f95f2e951f"
server_key_iv: "f67b2ce74e7db81a33a1911f11341ef7"
server_key_version: "encryptionkey@0"
I20260810 07:25:04.168853 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:04.177947 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:04.180418 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:04.180619 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:04.180807 2804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:04.181140 2804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:04.181236 2804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:04.181288 2804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:04.181314 2804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:04.187901 2804 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:46233
I20260810 07:25:04.188004 2948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:46233 every 8 connection(s)
I20260810 07:25:04.188441 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:04.197109 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.189256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:59701 (local address 127.24.60.254:42049)
0810 07:25:04.189328 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.189330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.189582 (+ 252us) server_negotiation.cc:407] Connection header received
0810 07:25:04.190194 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.190196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.190210 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.190273 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.191434 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.192003 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.193168 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.193403 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.193490 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.197042 (+ 3552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:59701: BlockingRecv error: recv got EOF from 127.24.60.195:59701 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:25:04.197227 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.189282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:59701)
0810 07:25:04.189446 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.189485 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.190069 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.190369 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.190373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.191062 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.191304 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.191309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.192113 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.192114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.193032 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.193034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.193515 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.193516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.193656 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.194238 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.194247 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.196951 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260810 07:25:04.197518 2950 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:42049 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.60.254:42049: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
I20260810 07:25:04.198635 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 2804
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:04.203630 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.197766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:58177)
0810 07:25:04.197823 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.197836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.197885 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.198144 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.198146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.198304 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.198359 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.198361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.199301 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.199302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.200008 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.200011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.200497 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.200498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.200579 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.201114 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.201120 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.203525 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 07:25:04.203688 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.197948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:58177 (local address 127.24.60.254:42049)
0810 07:25:04.198022 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.198024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.198031 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:25:04.198056 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.198057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.198066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.198137 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.198464 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.199191 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.200109 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.200400 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.200460 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.203612 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:58177: BlockingRecv error: recv got EOF from 127.24.60.195:58177 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:04.208281 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.199146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.24.60.254:42049)
0810 07:25:04.199320 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.199322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.199386 (+ 64us) server_negotiation.cc:407] Connection header received
0810 07:25:04.199464 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.199465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.199478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.199542 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.200235 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.200994 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.202169 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.202541 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.202627 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.204327 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:04.204334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:04.204334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:04.204349 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:04.206451 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:04.206945 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:04.206950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:04.206951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:04.207099 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:04.207338 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:04.207339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:04.207339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:04.207638 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:04.207800 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:04.207992 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 07:25:04.208129 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:04.209390 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.203989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:55265)
0810 07:25:04.204040 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.204051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.204096 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.204233 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.204233 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.204370 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.204403 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.204404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.205483 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.205484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.206034 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.206037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.206556 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.206557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.206621 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.207031 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.207036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.209312 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 07:25:04.209475 2648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.204042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:55265 (local address 127.24.60.254:42049)
0810 07:25:04.204084 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.204087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.204096 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:04.204165 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.204166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.204176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.204231 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.204498 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.205380 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.206130 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.206463 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.206545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.209406 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:55265: BlockingRecv error: recv got EOF from 127.24.60.195:55265 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 07:25:04.209578 2950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:04 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:04.944420 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:04.939213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:38683)
0810 07:25:04.939385 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:04.939399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:04.939464 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:04.939610 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:04.939611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:04.939860 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:04.939926 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.939929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.940844 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.940844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.941490 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:04.941494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.941832 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.941833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:04.941874 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.942404 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:04.942414 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:04.944352 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
W20260810 07:25:04.944571 2957 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:04.939323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:38683 (local address 127.24.60.254:42049)
0810 07:25:04.939479 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:04.939481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:04.939492 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:25:04.939519 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:04.939520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:04.939530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:04.939591 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:04.940057 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.940771 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.941568 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:04.941765 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:04.941822 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:04.944452 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:38683: BlockingRecv error: recv got EOF from 127.24.60.193:38683 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
Aug 10 07:25:05 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:05 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:05.084662 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.079131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:38589)
0810 07:25:05.079311 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.079326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.079437 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.079602 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.079604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.079804 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.079855 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.079856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.080653 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.080654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.081340 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.081343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.081729 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.081730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.081790 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.082323 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.082328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.084545 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260810 07:25:05.084704 2957 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:05.079240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:38589 (local address 127.24.60.254:42049)
0810 07:25:05.079316 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.079319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.079415 (+ 96us) server_negotiation.cc:407] Connection header received
0810 07:25:05.079503 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.079504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.079518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.079593 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.079949 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.080562 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.081432 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.081642 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.081728 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.084639 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:38589: BlockingRecv error: recv got EOF from 127.24.60.194:38589 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 07:25:05 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:05 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:05.215648 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.210050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:34549)
0810 07:25:05.210200 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.210236 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.210300 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.210456 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.210457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.210672 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.210724 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.210726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.211453 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.211454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.212010 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.212012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.212411 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.212412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.212531 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.213219 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.213225 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.215520 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
W20260810 07:25:05.215739 2957 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:05.210183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:34549 (local address 127.24.60.254:42049)
0810 07:25:05.210275 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.210277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.210288 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:25:05.210369 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.210370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.210384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.210438 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.210817 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.211335 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.212110 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.212324 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.212405 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.215670 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:34549: BlockingRecv error: recv got EOF from 127.24.60.195:34549 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 07:25:05 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:05 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:05.949944 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.945021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:38141)
0810 07:25:05.945234 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.945247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.945330 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.945494 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.945495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.945672 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.945723 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.945724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.946537 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.946538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.947097 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.947099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.947483 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.947484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.947526 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.948061 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.948066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.949884 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
W20260810 07:25:05.950106 2962 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:05.945092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:38141 (local address 127.24.60.254:42049)
0810 07:25:05.945295 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.945298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.945314 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:05.945391 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.945392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.945404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.945477 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.945861 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.946441 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.947197 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.947408 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.947476 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.950002 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:38141: BlockingRecv error: recv got EOF from 127.24.60.193:38141 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":133,"threads_started":1}
Aug 10 07:25:06 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:06 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:06.090942 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.085405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:40897)
0810 07:25:06.085608 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.085623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.085726 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.085889 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.085890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.086121 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:06.086173 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.086175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.086924 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.086925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.087575 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.087577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.087966 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.087967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.088036 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.088590 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:06.088595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:06.090868 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
W20260810 07:25:06.091091 2962 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:06.085535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:40897 (local address 127.24.60.254:42049)
0810 07:25:06.085612 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.085616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.085704 (+ 88us) server_negotiation.cc:407] Connection header received
0810 07:25:06.085797 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.085798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.085815 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.085881 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.086266 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.086819 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.087660 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.087871 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.087963 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.091002 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:40897: BlockingRecv error: recv got EOF from 127.24.60.194:40897 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 07:25:06 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:06 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:06.222375 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.216401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:57835)
0810 07:25:06.216574 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.216585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.216679 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.216916 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.216917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.217219 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:06.217283 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.217286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.218151 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.218152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.218956 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.218960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.219558 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.219559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.219642 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.220188 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:06.220195 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:06.222287 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
W20260810 07:25:06.222466 2962 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:06.216564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:57835 (local address 127.24.60.254:42049)
0810 07:25:06.216649 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.216651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.216666 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:06.216758 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.216760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.216775 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.216834 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.217392 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.218046 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.219088 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.219431 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.219521 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.222397 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:57835: BlockingRecv error: recv got EOF from 127.24.60.195:57835 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 07:25:06 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:06 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:06.960441 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.950477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:34495)
0810 07:25:06.950624 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.950637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.950700 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.950863 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.950864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.951051 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:06.951105 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.951106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.951771 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.951772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.952344 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.952347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.952689 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.952690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.952736 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.958460 (+ 5724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:06.958467 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:06.960368 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260810 07:25:06.960590 2966 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:06.950542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:34495 (local address 127.24.60.254:42049)
0810 07:25:06.950676 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.950678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.950687 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:06.950758 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.950759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.950770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.950831 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.951183 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.951689 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.952424 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.952617 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.952671 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.960495 (+ 7824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:34495: BlockingRecv error: recv got EOF from 127.24.60.193:34495 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
Aug 10 07:25:07 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:07 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:07.096694 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:07.091498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:58179)
0810 07:25:07.091689 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:07.091707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:07.091777 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:07.091960 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:07.091961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:07.092154 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:07.092207 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:07.092209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.092884 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.092886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:07.093544 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:07.093546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.094009 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.094009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:07.094100 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:07.094768 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:07.094773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:07.096633 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":138,"threads_started":1}
W20260810 07:25:07.096928 2966 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:07.091586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:58179 (local address 127.24.60.254:42049)
0810 07:25:07.091664 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:07.091666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:07.091779 (+ 113us) server_negotiation.cc:407] Connection header received
0810 07:25:07.091852 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:07.091853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:07.091866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:07.091937 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:07.092281 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.092785 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.093638 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.093906 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.094007 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:07.096840 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:58179: BlockingRecv error: recv got EOF from 127.24.60.194:58179 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 07:25:07 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:07 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:07.228551 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:07.223031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:39143)
0810 07:25:07.223524 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:07.223551 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:07.223654 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:07.223810 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:07.223811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:07.224014 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:07.224068 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:07.224070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.224783 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.224785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:07.225355 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:07.225356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.225760 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.225761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:07.225826 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:07.226397 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:07.226409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:07.228476 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":77,"threads_started":1}
W20260810 07:25:07.228659 2966 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:07.223138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:39143 (local address 127.24.60.254:42049)
0810 07:25:07.223201 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:07.223204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:07.223640 (+ 436us) server_negotiation.cc:407] Connection header received
0810 07:25:07.223720 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:07.223722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:07.223739 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:07.223801 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:07.224159 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.224679 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.225449 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.225664 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.225756 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:07.228581 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:39143: BlockingRecv error: recv got EOF from 127.24.60.195:39143 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 10 07:25:07 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:07 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:07.966204 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:07.961016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:37449)
0810 07:25:07.961222 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:07.961234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:07.961300 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:07.961444 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:07.961445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:07.961611 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:07.961662 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:07.961663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.962416 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.962416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:07.962986 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:07.962988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.963390 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.963391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:07.963440 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:07.963975 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:07.963980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:07.966141 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260810 07:25:07.966378 2971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:07.961097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:37449 (local address 127.24.60.254:42049)
0810 07:25:07.961249 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:07.961250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:07.961281 (+ 31us) server_negotiation.cc:407] Connection header received
0810 07:25:07.961355 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:07.961355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:07.961366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:07.961428 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:07.961733 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.962317 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.963084 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:07.963302 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:07.963372 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:07.966261 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:37449: BlockingRecv error: recv got EOF from 127.24.60.193:37449 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
Aug 10 07:25:08 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:08 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:08.102638 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:08.097282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:59701)
0810 07:25:08.097484 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:08.097507 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:08.097605 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:08.097764 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:08.097765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:08.097965 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:08.098019 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:08.098021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.098700 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.098701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:08.099234 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:08.099235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.099628 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.099629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:08.099698 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.100240 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:08.100245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:08.102538 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
W20260810 07:25:08.102766 2971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:08.097400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:59701 (local address 127.24.60.254:42049)
0810 07:25:08.097481 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:08.097483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:08.097587 (+ 104us) server_negotiation.cc:407] Connection header received
0810 07:25:08.097666 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:08.097667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:08.097680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:08.097757 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:08.098107 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.098614 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.099324 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.099538 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.099615 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.102678 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:59701: BlockingRecv error: recv got EOF from 127.24.60.194:59701 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 07:25:08 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:08 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:08.234359 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:08.229219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:56819)
0810 07:25:08.229393 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:08.229405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:08.229496 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:08.229654 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:08.229655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:08.229852 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:08.229903 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:08.229905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.230630 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.230632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:08.231187 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:08.231188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.231582 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.231583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:08.231648 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.232200 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:08.232205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:08.234296 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
W20260810 07:25:08.234516 2971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:08.229295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:56819 (local address 127.24.60.254:42049)
0810 07:25:08.229377 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:08.229380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:08.229481 (+ 101us) server_negotiation.cc:407] Connection header received
0810 07:25:08.229567 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:08.229568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:08.229582 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:08.229642 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:08.229987 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.230534 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.231281 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.231503 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.231583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.234425 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:56819: BlockingRecv error: recv got EOF from 127.24.60.195:56819 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 07:25:08 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:08 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:08.971578 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:08.966765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:53159)
0810 07:25:08.966911 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:08.966926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:08.966988 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:08.967153 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:08.967154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:08.967325 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:08.967391 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:08.967393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.968059 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.968060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:08.968594 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:08.968596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.969013 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.969014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:08.969056 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.969566 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:08.969571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:08.971518 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":136,"threads_started":1}
W20260810 07:25:08.971768 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:08.966867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53159 (local address 127.24.60.254:42049)
0810 07:25:08.967025 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:08.967027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:08.967036 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:08.967059 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:08.967060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:08.967071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:08.967135 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:08.967463 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.967972 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.968724 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.968938 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.969002 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.971641 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:53159: BlockingRecv error: recv got EOF from 127.24.60.193:53159 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
Aug 10 07:25:09 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:09 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:09.108711 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:09.103202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:60041)
0810 07:25:09.103386 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:09.103397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:09.103517 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:09.103708 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:09.103709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:09.103904 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:09.103953 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:09.103954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.104681 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.104682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:09.105298 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:09.105301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.105688 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.105689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:09.105752 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:09.106302 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:09.106307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:09.108604 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
W20260810 07:25:09.108777 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:09.103379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:60041 (local address 127.24.60.254:42049)
0810 07:25:09.103475 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:09.103477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:09.103488 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:25:09.103613 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:09.103614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:09.103628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:09.103698 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:09.104073 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.104586 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.105387 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.105597 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.105686 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:09.108697 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:60041: BlockingRecv error: recv got EOF from 127.24.60.194:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 07:25:09 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:09 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:09.240341 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:09.234974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:37843)
0810 07:25:09.235171 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:09.235185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:09.235284 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:09.235440 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:09.235441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:09.235647 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:09.235698 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:09.235700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.236421 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.236422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:09.236966 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:09.236968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.237442 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.237443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:09.237505 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:09.237973 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:09.237978 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:09.240229 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
W20260810 07:25:09.240406 2975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:09.235102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:37843 (local address 127.24.60.254:42049)
0810 07:25:09.235186 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:09.235188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:09.235260 (+ 72us) server_negotiation.cc:407] Connection header received
0810 07:25:09.235362 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:09.235363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:09.235379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:09.235434 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:09.235788 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.236327 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.237089 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.237365 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.237443 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:09.240322 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:37843: BlockingRecv error: recv got EOF from 127.24.60.195:37843 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 07:25:09 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:09 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:09.977682 2979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:09.972149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:34489)
0810 07:25:09.972318 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:09.972329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:09.972384 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:09.972604 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:09.972605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:09.972843 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:09.972907 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:09.972910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.973740 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.973742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:09.974374 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:09.974376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.974785 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.974786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:09.974905 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:09.975434 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:09.975439 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:09.977619 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260810 07:25:09.977861 2982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:09.972228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:34489 (local address 127.24.60.254:42049)
0810 07:25:09.972416 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:09.972418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:09.972428 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:25:09.972453 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:09.972454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:09.972465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:09.972536 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:09.972995 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.973639 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.974479 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:09.974692 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:09.974753 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:09.977751 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:34489: BlockingRecv error: recv got EOF from 127.24.60.193:34489 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":127,"threads_started":1}
Aug 10 07:25:10 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:10 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:10.119511 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:10.109540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:44673)
0810 07:25:10.109727 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:10.109747 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:10.109830 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:10.110030 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:10.110031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:10.110214 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:10.110268 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:10.110269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.111020 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.111021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:10.111617 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:10.111618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.111978 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.111979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:10.112022 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:10.117427 (+ 5405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:10.117434 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:10.119431 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
W20260810 07:25:10.119700 2982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:10.109673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:44673 (local address 127.24.60.254:42049)
0810 07:25:10.109749 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:10.109751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:10.109824 (+ 73us) server_negotiation.cc:407] Connection header received
0810 07:25:10.109888 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:10.109889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:10.109902 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:10.109972 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:10.110405 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.110932 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.111704 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.111908 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.111967 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:10.119621 (+ 7654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:44673: BlockingRecv error: recv got EOF from 127.24.60.194:44673 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 07:25:10 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:10 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:10.246636 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:10.240968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:59673)
0810 07:25:10.241226 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:10.241240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:10.241317 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:10.241639 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:10.241640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:10.241866 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:10.241937 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:10.241940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.242685 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.242686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:10.243289 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:10.243292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.243735 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.243736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:10.243833 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:10.244323 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:10.244329 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:10.246573 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
W20260810 07:25:10.246803 2982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:10.241049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:59673 (local address 127.24.60.254:42049)
0810 07:25:10.241212 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:10.241216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:10.241295 (+ 79us) server_negotiation.cc:407] Connection header received
0810 07:25:10.241382 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:10.241383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:10.241513 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:10.241625 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:10.242043 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.242574 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.243410 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.243620 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.243733 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:10.246715 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:59673: BlockingRecv error: recv got EOF from 127.24.60.195:59673 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 10 07:25:10 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:10 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:10.983244 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:10.978304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:38205)
0810 07:25:10.978497 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:10.978509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:10.978568 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:10.978741 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:10.978742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:10.978940 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:10.978990 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:10.978992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.979807 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.979808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:10.980361 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:10.980363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.980802 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.980803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:10.980842 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:10.981396 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:10.981415 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:10.983181 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":124,"threads_started":1}
W20260810 07:25:10.983492 2987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:10.978411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:38205 (local address 127.24.60.254:42049)
0810 07:25:10.978533 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:10.978536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:10.978551 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:10.978618 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:10.978619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:10.978631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:10.978718 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:10.979059 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.979712 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.980459 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:10.980734 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:10.980792 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:10.983408 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:38205: BlockingRecv error: recv got EOF from 127.24.60.193:38205 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
Aug 10 07:25:11 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:11 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:11.125010 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:11.120023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:50097)
0810 07:25:11.120195 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:11.120211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:11.120266 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:11.120434 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:11.120435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:11.120609 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:11.120667 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:11.120669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.121482 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.121483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:11.122019 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:11.122021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.122428 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.122429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:11.122475 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:11.123003 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:11.123008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:11.124950 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
W20260810 07:25:11.125198 2987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:11.120115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:50097 (local address 127.24.60.254:42049)
0810 07:25:11.120204 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:11.120207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:11.120256 (+ 49us) server_negotiation.cc:407] Connection header received
0810 07:25:11.120349 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:11.120350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:11.120361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:11.120413 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:11.120769 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.121391 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.122111 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.122350 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.122417 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:11.125062 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:50097: BlockingRecv error: recv got EOF from 127.24.60.194:50097 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 07:25:11 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:11 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:11.252295 2989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:11.247212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:34383)
0810 07:25:11.247363 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:11.247375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:11.247435 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:11.247579 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:11.247579 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:11.247764 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:11.247819 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:11.247821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.248509 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.248510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:11.249044 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:11.249046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.249527 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.249528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:11.249572 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:11.250089 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:11.250094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:11.252206 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
W20260810 07:25:11.252420 2987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:11.247327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:34383 (local address 127.24.60.254:42049)
0810 07:25:11.247398 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:11.247404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:11.247448 (+ 44us) server_negotiation.cc:407] Connection header received
0810 07:25:11.247499 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:11.247500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:11.247512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:11.247558 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:11.247907 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.248425 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.249235 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.249451 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.249526 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:11.252345 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:34383: BlockingRecv error: recv got EOF from 127.24.60.195:34383 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 07:25:11 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:11 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:11.994520 2990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:11.983953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:50163)
0810 07:25:11.984141 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:11.984157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:11.984246 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:11.984426 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:11.984427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:11.984604 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:11.984658 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:11.984660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.985662 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.985663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:11.986257 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:11.986259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.986704 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.986704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:11.986754 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:11.992193 (+ 5439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:11.992200 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:11.994455 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260810 07:25:11.994709 2991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:11.984122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:50163 (local address 127.24.60.254:42049)
0810 07:25:11.984270 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:11.984272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:11.984281 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:11.984308 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:11.984309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:11.984320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:11.984383 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:11.984866 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.985550 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.986373 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:11.986617 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:11.986683 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:11.994625 (+ 7942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:50163: BlockingRecv error: recv got EOF from 127.24.60.193:50163 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
Aug 10 07:25:12 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:12 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:12.130792 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.125634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:36919)
0810 07:25:12.125816 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:12.125826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:12.125886 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:12.126052 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:12.126053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:12.126232 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:12.126286 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.126287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.127138 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.127140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.127707 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.127709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.128116 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.128117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.128169 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.128728 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:12.128733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:12.130728 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
W20260810 07:25:12.131048 2991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:12.125739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:36919 (local address 127.24.60.254:42049)
0810 07:25:12.125819 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.125821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.125888 (+ 67us) server_negotiation.cc:407] Connection header received
0810 07:25:12.125958 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.125958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.125970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.126026 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.126357 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.127047 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.127796 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.128013 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.128070 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.130959 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:36919: BlockingRecv error: recv got EOF from 127.24.60.194:36919 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 07:25:12 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:12 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:12.258275 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.252943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:37125)
0810 07:25:12.253090 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:12.253102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:12.253221 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:12.253415 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:12.253416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:12.253590 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:12.253643 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.253645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.254365 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.254366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.254912 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.254914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.255272 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.255273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.255318 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.255885 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:12.255891 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:12.258172 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260810 07:25:12.258335 2991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:12.252973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:37125 (local address 127.24.60.254:42049)
0810 07:25:12.253059 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.253061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.253199 (+ 138us) server_negotiation.cc:407] Connection header received
0810 07:25:12.253298 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.253299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.253313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.253378 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.253730 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.254271 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.255003 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.255197 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.255255 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.258271 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:37125: BlockingRecv error: recv got EOF from 127.24.60.195:37125 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 07:25:12 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:12 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:12.999886 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.995086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:41057)
0810 07:25:12.995248 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:12.995260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:12.995316 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:12.995458 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:12.995459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:12.995657 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:12.995708 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.995709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.996415 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.996416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.996947 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.996948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.997314 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.997315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.997361 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.997889 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:12.997894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:12.999826 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
W20260810 07:25:13.000176 2996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:12.995153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:41057 (local address 127.24.60.254:42049)
0810 07:25:12.995312 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.995314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.995327 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:12.995355 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.995356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.995367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.995439 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.995777 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.996332 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.997029 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.997254 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.997301 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.999958 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:41057: BlockingRecv error: recv got EOF from 127.24.60.193:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Aug 10 07:25:13 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:13 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:13.137053 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:13.131349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:47635)
0810 07:25:13.131515 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:13.131527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:13.131597 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:13.131766 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:13.131767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:13.132003 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:13.132053 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:13.132055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.132824 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.132825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:13.133482 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:13.133485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.133893 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.133895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:13.133965 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:13.134453 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:13.134458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:13.136951 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
W20260810 07:25:13.137106 2996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:13.131534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:47635 (local address 127.24.60.254:42049)
0810 07:25:13.131623 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:13.131626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:13.131637 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:25:13.131668 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:13.131669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:13.131683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:13.131758 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:13.132175 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.132733 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.133574 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.133776 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.133871 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:13.137037 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:47635: BlockingRecv error: recv got EOF from 127.24.60.194:47635 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 07:25:13 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:13 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:13.264489 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:13.258913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:47169)
0810 07:25:13.259089 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:13.259103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:13.259205 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:13.259437 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:13.259438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:13.259636 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:13.259690 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:13.259691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.260354 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.260356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:13.260941 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:13.260943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.261462 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.261463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:13.261540 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:13.262064 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:13.262070 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:13.264425 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
W20260810 07:25:13.264623 2996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:13.259030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:47169 (local address 127.24.60.254:42049)
0810 07:25:13.259080 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:13.259082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:13.259189 (+ 107us) server_negotiation.cc:407] Connection header received
0810 07:25:13.259318 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:13.259319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:13.259336 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:13.259424 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:13.259758 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.260271 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.261056 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:13.261369 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:13.261436 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:13.264536 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:47169: BlockingRecv error: recv got EOF from 127.24.60.195:47169 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 07:25:13 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:14 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:14.006335 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:14.000679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:48309)
0810 07:25:14.001025 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:14.001037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:14.001110 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:14.001308 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:14.001308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:14.001501 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:14.001554 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:14.001555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.002320 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.002321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:14.002943 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:14.002946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.003561 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.003563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:14.003628 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.004133 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:14.004138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:14.006276 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":252,"threads_started":1}
W20260810 07:25:14.006500 3001 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:14.000847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:48309 (local address 127.24.60.254:42049)
0810 07:25:14.001034 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:14.001036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:14.001118 (+ 82us) server_negotiation.cc:407] Connection header received
0810 07:25:14.001206 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:14.001207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:14.001221 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:14.001298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:14.001658 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.002228 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.003082 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.003443 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.003531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.006404 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:48309: BlockingRecv error: recv got EOF from 127.24.60.193:48309 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
Aug 10 07:25:14 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:14 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:14.143100 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:14.137636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:53815)
0810 07:25:14.137843 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:14.137857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:14.137954 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:14.138117 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:14.138118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:14.138309 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:14.138359 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:14.138361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.139187 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.139188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:14.139887 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:14.139891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.140345 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.140346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:14.140409 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.140962 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:14.140967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:14.143004 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":135,"threads_started":1}
W20260810 07:25:14.143169 3001 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:14.137750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:53815 (local address 127.24.60.254:42049)
0810 07:25:14.137844 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:14.137846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:14.137944 (+ 98us) server_negotiation.cc:407] Connection header received
0810 07:25:14.138032 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:14.138033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:14.138045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:14.138104 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:14.138489 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.139114 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.139997 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.140271 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.140345 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.143105 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:53815: BlockingRecv error: recv got EOF from 127.24.60.194:53815 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 07:25:14 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:14 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:14.270370 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:14.265265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:41765)
0810 07:25:14.265516 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:14.265532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:14.265621 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:14.265789 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:14.265789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:14.266010 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:14.266060 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:14.266062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.266800 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.266801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:14.267344 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:14.267345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.267746 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.267747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:14.267813 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.268288 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:14.268293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:14.270305 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":127,"threads_started":1}
W20260810 07:25:14.270517 3001 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:14.265450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:41765 (local address 127.24.60.254:42049)
0810 07:25:14.265534 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:14.265537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:14.265599 (+ 62us) server_negotiation.cc:407] Connection header received
0810 07:25:14.265685 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:14.265686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:14.265701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:14.265780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:14.266177 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.266708 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.267451 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.267667 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.267745 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.270421 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:41765: BlockingRecv error: recv got EOF from 127.24.60.195:41765 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 07:25:15 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:15 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:15.013087 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:15.007153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:57799)
0810 07:25:15.007327 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:15.007347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:15.007413 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:15.007630 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:15.007631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:15.007889 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:15.007950 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:15.007952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.008970 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.008972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:15.009810 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:15.009814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.010253 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.010255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:15.010327 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:15.010894 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:15.010900 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:15.012956 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
W20260810 07:25:15.013144 3006 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:15.007260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:57799 (local address 127.24.60.254:42049)
0810 07:25:15.007449 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:15.007452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:15.007472 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:25:15.007502 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:15.007502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:15.007514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:15.007604 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:15.008044 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.008767 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.009921 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.010162 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.010249 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:15.013058 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:57799: BlockingRecv error: recv got EOF from 127.24.60.193:57799 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":125,"threads_started":1}
Aug 10 07:25:15 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:15 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:15.148794 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:15.143589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:58225)
0810 07:25:15.143779 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:15.143789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:15.143843 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:15.143994 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:15.143994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:15.144183 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:15.144234 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:15.144235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.144978 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.144979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:15.145597 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:15.145599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.146054 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.146055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:15.146096 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:15.146637 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:15.146647 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:15.148723 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
W20260810 07:25:15.148937 3006 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:15.143672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:58225 (local address 127.24.60.254:42049)
0810 07:25:15.143754 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:15.143756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:15.143832 (+ 76us) server_negotiation.cc:407] Connection header received
0810 07:25:15.143895 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:15.143896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:15.143908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:15.143967 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:15.144305 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.144849 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.145707 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.145974 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.146049 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:15.148837 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:58225: BlockingRecv error: recv got EOF from 127.24.60.194:58225 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 07:25:15 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:15 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:15.276692 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:15.271075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:58877)
0810 07:25:15.271295 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:15.271309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:15.271430 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:15.271602 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:15.271603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:15.271805 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:15.271858 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:15.271860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.272583 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.272584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:15.273225 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:15.273229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.273611 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.273612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:15.273680 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:15.274149 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:15.274154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:15.276607 (+ 2453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":121,"threads_started":1}
W20260810 07:25:15.276904 3006 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:15.271188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:58877 (local address 127.24.60.254:42049)
0810 07:25:15.271296 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:15.271299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:15.271428 (+ 129us) server_negotiation.cc:407] Connection header received
0810 07:25:15.271500 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:15.271501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:15.271516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:15.271594 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:15.271960 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.272478 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.273314 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:15.273521 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:15.273599 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:15.276725 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:58877: BlockingRecv error: recv got EOF from 127.24.60.195:58877 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 07:25:16 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:16 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:16.018847 3011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:16.013842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:33473 (local address 127.24.60.254:42049)
0810 07:25:16.013981 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.013983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.013992 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:16.014018 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.014019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.014030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.014102 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.014465 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.015032 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.015780 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.016006 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.016069 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.018776 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:33473: BlockingRecv error: recv got EOF from 127.24.60.193:33473 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260810 07:25:16.018869 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.013689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:33473)
0810 07:25:16.013842 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:16.013851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:16.013905 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:16.014128 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:16.014129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:16.014311 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:16.014361 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.014363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.015125 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.015126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.015687 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.015688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.016084 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.016085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.016130 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.016631 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:16.016636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:16.018688 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
Aug 10 07:25:16 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:16 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:16.154443 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.149407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:58399)
0810 07:25:16.149601 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:16.149613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:16.149716 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:16.149897 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:16.149898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:16.150128 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:16.150177 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.150179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.150883 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.150885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.151424 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.151426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.151788 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.151789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.151848 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.152469 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:16.152475 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:16.154374 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
W20260810 07:25:16.154608 3011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:16.149508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:58399 (local address 127.24.60.254:42049)
0810 07:25:16.149598 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.149600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.149696 (+ 96us) server_negotiation.cc:407] Connection header received
0810 07:25:16.149801 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.149802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.149815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.149889 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.150274 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.150783 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.151515 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.151713 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.151790 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.154518 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:58399: BlockingRecv error: recv got EOF from 127.24.60.194:58399 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 07:25:16 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:16 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:16.282275 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.277285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:35975)
0810 07:25:16.277448 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:16.277466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:16.277528 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:16.277667 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:16.277667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:16.277838 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:16.277890 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.277891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.278591 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.278592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.279152 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.279154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.279502 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.279503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.279544 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.280035 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:16.280039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:16.282173 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260810 07:25:16.282330 3011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:16.277367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:35975 (local address 127.24.60.254:42049)
0810 07:25:16.277444 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.277447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.277527 (+ 80us) server_negotiation.cc:407] Connection header received
0810 07:25:16.277583 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.277584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.277595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.277650 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.277999 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.278516 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.279233 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.279437 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.279491 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.282261 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:35975: BlockingRecv error: recv got EOF from 127.24.60.195:35975 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 07:25:17 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:17 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:17.027809 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:17.019438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:43963)
0810 07:25:17.019594 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:17.019610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:17.019697 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:17.019856 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:17.019857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:17.020074 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:17.020123 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:17.020124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.020798 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.020798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:17.021374 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:17.021376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.021726 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.021727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:17.021767 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.025551 (+ 3784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:17.025556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:17.027733 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
W20260810 07:25:17.027956 3015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:17.019556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:43963 (local address 127.24.60.254:42049)
0810 07:25:17.019724 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:17.019726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:17.019734 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:25:17.019758 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:17.019759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:17.019770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:17.019830 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:17.020203 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.020727 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.021464 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.021657 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.021713 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.027870 (+ 6157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:43963: BlockingRecv error: recv got EOF from 127.24.60.193:43963 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
Aug 10 07:25:17 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:17 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:17.159986 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:17.154975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:40255)
0810 07:25:17.155137 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:17.155149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:17.155214 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:17.155379 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:17.155380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:17.155574 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:17.155626 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:17.155627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.156367 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.156369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:17.156970 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:17.156973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.157347 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.157348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:17.157395 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.157990 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:17.157996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:17.159905 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
W20260810 07:25:17.160149 3015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:17.155159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:40255 (local address 127.24.60.254:42049)
0810 07:25:17.155239 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:17.155241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:17.155252 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:25:17.155291 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:17.155292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:17.155303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:17.155357 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:17.155728 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.156273 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.157059 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.157281 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.157337 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.160077 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:40255: BlockingRecv error: recv got EOF from 127.24.60.194:40255 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 07:25:17 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:17 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:17.288045 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:17.282801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:43527)
0810 07:25:17.282969 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:17.283003 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:17.283088 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:17.283277 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:17.283278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:17.283476 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:17.283525 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:17.283526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.284224 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.284229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:17.284780 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:17.284782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.285185 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.285186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:17.285244 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.285767 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:17.285772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:17.287878 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260810 07:25:17.288030 3015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:17.282958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:43527 (local address 127.24.60.254:42049)
0810 07:25:17.283036 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:17.283039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:17.283065 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:25:17.283176 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:17.283177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:17.283198 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:17.283257 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:17.283606 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.284113 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.284872 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.285074 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.285205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.287970 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:43527: BlockingRecv error: recv got EOF from 127.24.60.195:43527 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 07:25:18 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:18 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:18.033988 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:18.028375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:43683)
0810 07:25:18.028553 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:18.028564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:18.028619 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:18.028835 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:18.028836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:18.029022 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:18.029079 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:18.029081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.029850 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.029851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:18.030462 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:18.030464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.030854 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.030855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:18.030907 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.031415 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:18.031420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:18.033837 (+ 2417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
W20260810 07:25:18.034009 3019 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:18.028502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:43683 (local address 127.24.60.254:42049)
0810 07:25:18.028645 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:18.028648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:18.028656 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:25:18.028680 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:18.028681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:18.028692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:18.028764 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:18.029195 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.029733 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.030551 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.030754 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.030812 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.033941 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:43683: BlockingRecv error: recv got EOF from 127.24.60.193:43683 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Aug 10 07:25:18 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:18 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:18.165750 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:18.160528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:37701)
0810 07:25:18.160736 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:18.160749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:18.160847 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:18.161012 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:18.161013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:18.161250 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:18.161320 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:18.161322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.162030 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.162031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:18.162581 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:18.162583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.162931 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.162932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:18.162984 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.163580 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:18.163585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:18.165621 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
W20260810 07:25:18.165930 3019 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:18.160668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:37701 (local address 127.24.60.254:42049)
0810 07:25:18.160737 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:18.160739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:18.160849 (+ 110us) server_negotiation.cc:407] Connection header received
0810 07:25:18.160912 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:18.160913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:18.160925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:18.160982 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:18.161401 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.161927 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.162665 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.162849 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.162903 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.165786 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:37701: BlockingRecv error: recv got EOF from 127.24.60.194:37701 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 07:25:18 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:18 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.195@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:18.293711 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:18.288578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.195:56857)
0810 07:25:18.288787 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:18.288799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:18.288872 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:18.289066 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:18.289067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:18.289280 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:18.289328 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:18.289330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.290077 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.290078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:18.290641 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:18.290642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.291025 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.291026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:18.291096 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.291604 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:18.291609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:18.293612 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
W20260810 07:25:18.293769 3019 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:18.288624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:56857 (local address 127.24.60.254:42049)
0810 07:25:18.288692 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:18.288694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:18.288858 (+ 164us) server_negotiation.cc:407] Connection header received
0810 07:25:18.288962 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:18.288963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:18.288977 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:18.289054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:18.289447 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.289990 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.290733 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.290947 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.291026 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.293703 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.195:56857: BlockingRecv error: recv got EOF from 127.24.60.195:56857 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 07:25:19 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:19 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.193@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:19.039980 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.034600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.193:54431)
0810 07:25:19.034757 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.034767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.034825 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.034999 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.035000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.035175 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.035227 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.035228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.035937 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.035939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.036838 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.036841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.037316 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.037318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.037369 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.037828 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:19.037833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.039843 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260810 07:25:19.039976 3024 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:19.034680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:54431 (local address 127.24.60.254:42049)
0810 07:25:19.034835 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.034838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.034882 (+ 44us) server_negotiation.cc:407] Connection header received
0810 07:25:19.034909 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.034910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.034921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.034978 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.035301 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.035828 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.036945 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.037227 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.037287 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.039900 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.193:54431: BlockingRecv error: recv got EOF from 127.24.60.193:54431 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
Aug 10 07:25:19 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:19 dist-test-slave-vxj2 krb5kdc[1798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.60.194@KRBTEST.COM for oryx/127.24.60.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:19.171958 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.166409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42049 (local address 127.24.60.194:42355)
0810 07:25:19.166643 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.166656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.166727 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.166932 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.166933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.167110 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.167163 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.167164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.167889 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.167890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.168438 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.168439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.168850 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.168850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.168929 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.169555 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:19.169561 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.171868 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.60.254:42049: Server oryx/127.24.60.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":136,"threads_started":1}
W20260810 07:25:19.172091 3024 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:19.166595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:42355 (local address 127.24.60.254:42049)
0810 07:25:19.166670 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.166672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.166713 (+ 41us) server_negotiation.cc:407] Connection header received
0810 07:25:19.166780 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.166781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.166792 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.166885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.167256 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.167779 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.168547 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.168754 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.168848 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.172028 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.60.194:42355: BlockingRecv error: recv got EOF from 127.24.60.194:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:25:19.200934 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 2473
I20260810 07:25:19.211416 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 2649
I20260810 07:25:19.218182 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 2804
I20260810 07:25:19.224884 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 2394
2026-08-10T07:25:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17708 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T07:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:19Z Disabled control of system clock
I20260810 07:25:19.255826 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33963
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 07:25:19.342308 3030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:19.342542 3030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:19.342564 3030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:19.343838 3030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:19.343886 3030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:19.343899 3030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:19.343911 3030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:19.343923 3030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:19.345395 3030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:19.345687 3030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:19.346028 3030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:19.351514 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.351522 3036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.351527 3035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:19.351627 3030 server_base.cc:1034] running on GCE node
I20260810 07:25:19.352042 3030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:19.352331 3030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:19.353471 3030 hybrid_clock.cc:648] HybridClock initialized: now 1786346719353457 us; error 36 us; skew 500 ppm
I20260810 07:25:19.354809 3030 webserver.cc:460] Webserver started at http://127.24.60.254:37147/ using document root <none> and password file <none>
I20260810 07:25:19.355118 3030 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:19.355168 3030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:19.355352 3030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:19.356523 3030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "9944de9ce30d48e3b1e320be51641bb7"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "4356b52e281728b16f614a22f4c56aaf"
server_key_iv: "2e5f65ef1c7ce21161a74acc4dd6cb44"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.356909 3030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "9944de9ce30d48e3b1e320be51641bb7"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "4356b52e281728b16f614a22f4c56aaf"
server_key_iv: "2e5f65ef1c7ce21161a74acc4dd6cb44"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.358443 3030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:25:19.359308 3047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.359529 3030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:19.359638 3030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "9944de9ce30d48e3b1e320be51641bb7"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "4356b52e281728b16f614a22f4c56aaf"
server_key_iv: "2e5f65ef1c7ce21161a74acc4dd6cb44"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.359730 3030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:19.384465 3030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:19.384907 3030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:19.385100 3030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:19.390779 3030 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:33963
I20260810 07:25:19.390841 3109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:33963 every 8 connection(s)
I20260810 07:25:19.391300 3030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:19.392164 3110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:19.395213 3110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7: Bootstrap starting.
I20260810 07:25:19.395917 3110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:19.396257 3110 log.cc:824] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:19.397125 3110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7: No bootstrap required, opened a new log
I20260810 07:25:19.398873 3110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944de9ce30d48e3b1e320be51641bb7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33963 } }
I20260810 07:25:19.399029 3110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:19.399053 3110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9944de9ce30d48e3b1e320be51641bb7, State: Initialized, Role: FOLLOWER
I20260810 07:25:19.399224 3110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944de9ce30d48e3b1e320be51641bb7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33963 } }
I20260810 07:25:19.399317 3110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:19.399360 3110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:19.399415 3110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:19.400033 3110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944de9ce30d48e3b1e320be51641bb7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33963 } }
I20260810 07:25:19.400151 3110 leader_election.cc:302] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9944de9ce30d48e3b1e320be51641bb7; no voters:
I20260810 07:25:19.400352 3110 leader_election.cc:288] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:19.400405 3113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:19.400658 3110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:19.400640 3113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 9944de9ce30d48e3b1e320be51641bb7, State: Running, Role: LEADER
I20260810 07:25:19.400821 3113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944de9ce30d48e3b1e320be51641bb7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33963 } }
I20260810 07:25:19.401190 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3030
I20260810 07:25:19.401306 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:25:19.401288 3114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9944de9ce30d48e3b1e320be51641bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944de9ce30d48e3b1e320be51641bb7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33963 } } }
I20260810 07:25:19.401374 3114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:19.401546 24819 external_mini_cluster.cc:1428] Setting key 697c9f04023d029b454b6008deef4085
I20260810 07:25:19.402012 3116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9944de9ce30d48e3b1e320be51641bb7. Latest consensus state: current_term: 1 leader_uuid: "9944de9ce30d48e3b1e320be51641bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944de9ce30d48e3b1e320be51641bb7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33963 } } }
I20260810 07:25:19.402103 3116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7 [sys.catalog]: This master's current role is: LEADER
W20260810 07:25:19.402781 3134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:25:19.402841 3134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:25:19.402978 3124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:19.403193 3124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:19.404913 3124 catalog_manager.cc:1347] Generated new cluster ID: 94a9e4f68ecc429cb06b032a43634878
I20260810 07:25:19.404978 3124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:19.407732 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.402292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.24.60.254:33963)
0810 07:25:19.402515 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.402526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.402717 (+ 191us) server_negotiation.cc:407] Connection header received
0810 07:25:19.402851 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.402859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.402942 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.403124 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.403481 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.404712 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.405721 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.406148 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.406287 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.407055 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.407059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.407060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:19.407248 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.407281 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:19.407293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.407387 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.407467 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.407555 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260810 07:25:19.455277 3124 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:19.456064 3124 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:19.465437 3124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9944de9ce30d48e3b1e320be51641bb7: Generated new TSK 0
I20260810 07:25:19.465677 3124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:19.469398 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33963
--builtin_ntp_servers=127.24.60.212:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 07:25:19.554448 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:19.554689 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:19.554713 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:19.555995 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:19.556042 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:19.556068 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:25:19.557530 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:19.557790 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:19.558109 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:19.563657 3145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.563668 3146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.563678 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:19.564536 3138 server_base.cc:1034] running on GCE node
I20260810 07:25:19.564797 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:19.565076 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:19.566228 3138 hybrid_clock.cc:648] HybridClock initialized: now 1786346719566215 us; error 38 us; skew 500 ppm
I20260810 07:25:19.567564 3138 webserver.cc:460] Webserver started at http://127.24.60.193:33025/ using document root <none> and password file <none>
I20260810 07:25:19.567829 3138 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:19.567934 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:19.568100 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:19.569331 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "50c05016621f41bc93d0902fa8749ce8"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "29e158a7eb2f3da26ea714b12e4eb3ae"
server_key_iv: "21cbeef289eab771666ddb9148dfb82e"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.569723 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "50c05016621f41bc93d0902fa8749ce8"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "29e158a7eb2f3da26ea714b12e4eb3ae"
server_key_iv: "21cbeef289eab771666ddb9148dfb82e"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.571120 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:19.571951 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.572152 3138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:19.572268 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "50c05016621f41bc93d0902fa8749ce8"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "29e158a7eb2f3da26ea714b12e4eb3ae"
server_key_iv: "21cbeef289eab771666ddb9148dfb82e"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.572345 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:19.585419 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:19.585772 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:19.585937 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:19.586187 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:19.586591 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:19.586652 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.586694 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:19.586723 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.592757 3138 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:33211
I20260810 07:25:19.592820 3289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:33211 every 8 connection(s)
I20260810 07:25:19.593287 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:19.595160 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3138
I20260810 07:25:19.595263 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:19.595484 24819 external_mini_cluster.cc:1428] Setting key 03cb728dc1051788448d3e9b04649984
I20260810 07:25:19.596577 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33963
--builtin_ntp_servers=127.24.60.212:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 07:25:19.599907 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.594129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:35741 (local address 127.24.60.254:33963)
0810 07:25:19.594219 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.594223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.594424 (+ 201us) server_negotiation.cc:407] Connection header received
0810 07:25:19.594967 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.594968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.594982 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.595038 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.596013 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.596597 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.598228 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.598566 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.598648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.599601 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.599603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.599603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:19.599608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.599622 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:19.599626 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.599674 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.599779 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.599813 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:25:19.600174 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.594077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33963 (local address 127.24.60.193:35741)
0810 07:25:19.594290 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.594325 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.594834 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.595127 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.595132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.595145 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.595915 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.595924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.596730 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.596731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.598104 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.598108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.598663 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.598664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.598779 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.599428 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:19.599429 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.599470 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:19.599500 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:19.599692 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:19.599699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:19.599703 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:19.599773 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 07:25:19.599894 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260810 07:25:19.600771 3290 heartbeater.cc:347] Connected to a master server at 127.24.60.254:33963
I20260810 07:25:19.600888 3290 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:19.601133 3290 heartbeater.cc:511] Master 127.24.60.254:33963 requested a full tablet report, sending...
I20260810 07:25:19.601809 3069 ts_manager.cc:194] Registered new tserver with Master: 50c05016621f41bc93d0902fa8749ce8 (127.24.60.193:33211)
I20260810 07:25:19.602479 3069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.193:35741
W20260810 07:25:19.682862 3293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:19.683087 3293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:19.683111 3293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:19.684396 3293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:19.684445 3293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:19.684473 3293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:25:19.686125 3293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:19.686399 3293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:19.686750 3293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:19.692266 3299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.692319 3302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.692324 3300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:19.692531 3293 server_base.cc:1034] running on GCE node
I20260810 07:25:19.692835 3293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:19.693117 3293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:19.694312 3293 hybrid_clock.cc:648] HybridClock initialized: now 1786346719694291 us; error 45 us; skew 500 ppm
I20260810 07:25:19.695717 3293 webserver.cc:460] Webserver started at http://127.24.60.194:35899/ using document root <none> and password file <none>
I20260810 07:25:19.695998 3293 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:19.696079 3293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:19.696244 3293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:19.697486 3293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "7935c9d5e7114bdcb5e6ea77eb9d0be7"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "092216bb1ad97956e923f341b23a9f6e"
server_key_iv: "2f3f1e92afd346267bbbf80653423ae3"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.697873 3293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "7935c9d5e7114bdcb5e6ea77eb9d0be7"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "092216bb1ad97956e923f341b23a9f6e"
server_key_iv: "2f3f1e92afd346267bbbf80653423ae3"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.699333 3293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 07:25:19.700346 3308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.700577 3293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:19.700696 3293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "7935c9d5e7114bdcb5e6ea77eb9d0be7"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "092216bb1ad97956e923f341b23a9f6e"
server_key_iv: "2f3f1e92afd346267bbbf80653423ae3"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.700790 3293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:19.722208 3293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:19.722595 3293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:19.722769 3293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:19.723038 3293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:19.723421 3293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:19.723479 3293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.723515 3293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:19.723562 3293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.729888 3293 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:42937
I20260810 07:25:19.729976 3438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:42937 every 8 connection(s)
I20260810 07:25:19.730358 3293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:19.733548 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3293
I20260810 07:25:19.733666 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:19.733899 24819 external_mini_cluster.cc:1428] Setting key 23083c9130f3537cc309d96b9810b544
I20260810 07:25:19.735234 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33963
--builtin_ntp_servers=127.24.60.212:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 07:25:19.736857 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.731251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:37233 (local address 127.24.60.254:33963)
0810 07:25:19.731322 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.731324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.731575 (+ 251us) server_negotiation.cc:407] Connection header received
0810 07:25:19.732084 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.732084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.732095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.732159 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.732896 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.733581 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.734988 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.735305 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.735396 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.736543 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.736544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.736545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:19.736551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.736571 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:19.736576 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.736644 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.736765 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.736795 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:25:19.737195 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.731195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33963 (local address 127.24.60.194:37233)
0810 07:25:19.731433 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.731484 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.731990 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.732218 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.732221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.732229 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.732795 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.732801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.733698 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.733700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.734901 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.734905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.735550 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.735551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.735683 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.736314 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:19.736314 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.736348 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:19.736374 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:19.736659 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:19.736668 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:19.736672 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:19.736753 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 07:25:19.736889 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260810 07:25:19.737826 3439 heartbeater.cc:347] Connected to a master server at 127.24.60.254:33963
I20260810 07:25:19.737938 3439 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:19.738145 3439 heartbeater.cc:511] Master 127.24.60.254:33963 requested a full tablet report, sending...
I20260810 07:25:19.738550 3069 ts_manager.cc:194] Registered new tserver with Master: 7935c9d5e7114bdcb5e6ea77eb9d0be7 (127.24.60.194:42937)
I20260810 07:25:19.738948 3069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.194:37233
W20260810 07:25:19.821359 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:19.821599 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:19.821623 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:19.822881 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:19.822932 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:19.822959 3443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:25:19.824394 3443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:19.824688 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:19.825016 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:19.830312 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.830416 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.830423 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:19.830603 3443 server_base.cc:1034] running on GCE node
I20260810 07:25:19.830809 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:19.831080 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:19.832253 3443 hybrid_clock.cc:648] HybridClock initialized: now 1786346719832240 us; error 44 us; skew 500 ppm
I20260810 07:25:19.833683 3443 webserver.cc:460] Webserver started at http://127.24.60.195:32957/ using document root <none> and password file <none>
I20260810 07:25:19.833972 3443 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:19.834049 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:19.834206 3443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:19.835391 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "b1191501db254ccc9572bf0d6079c796"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "4a447933ebd21d23904bfba8f76672c6"
server_key_iv: "9b93bbd2f3b052a53259b208763d2766"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.835801 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "b1191501db254ccc9572bf0d6079c796"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "4a447933ebd21d23904bfba8f76672c6"
server_key_iv: "9b93bbd2f3b052a53259b208763d2766"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.837270 3443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:25:19.838069 3461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.838249 3443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:19.838356 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "b1191501db254ccc9572bf0d6079c796"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-vxj2"
server_key: "4a447933ebd21d23904bfba8f76672c6"
server_key_iv: "9b93bbd2f3b052a53259b208763d2766"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.838444 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:19.857963 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:19.858338 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:19.858520 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:19.858783 3443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:19.859160 3443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:19.859217 3443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.859252 3443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:19.859297 3443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.865929 3443 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:45719
I20260810 07:25:19.866006 3595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:45719 every 8 connection(s)
I20260810 07:25:19.866387 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:19.871840 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3443
I20260810 07:25:19.871984 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:19.872185 24819 external_mini_cluster.cc:1428] Setting key 606e5319c1f83709ba61d182dd4c58ec
I20260810 07:25:19.872339 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.867284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:40903 (local address 127.24.60.254:33963)
0810 07:25:19.867413 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.867416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.867641 (+ 225us) server_negotiation.cc:407] Connection header received
0810 07:25:19.868265 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.868266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.868279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.868348 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.869070 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.869656 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.870726 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.870950 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.871041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.872034 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.872035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.872036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:19.872041 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.872057 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:19.872063 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.872115 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.872220 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.872256 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:25:19.872608 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.867239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33963 (local address 127.24.60.195:40903)
0810 07:25:19.867475 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.867529 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.868143 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.868426 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.868431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.868441 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.868966 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.868971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.869761 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.869762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.870625 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.870628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.871055 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.871056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.871204 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.871874 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:19.871874 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.871906 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:19.871930 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:19.872115 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:19.872123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:19.872127 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:19.872221 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:25:19.872334 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260810 07:25:19.873250 3597 heartbeater.cc:347] Connected to a master server at 127.24.60.254:33963
I20260810 07:25:19.873382 3597 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:19.873584 3597 heartbeater.cc:511] Master 127.24.60.254:33963 requested a full tablet report, sending...
I20260810 07:25:19.874065 3068 ts_manager.cc:194] Registered new tserver with Master: b1191501db254ccc9572bf0d6079c796 (127.24.60.195:45719)
I20260810 07:25:19.874473 3068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.195:40903
I20260810 07:25:19.884768 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 07:25:19.887007 3135 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:19.886431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.24.60.254:33963)
0810 07:25:19.886533 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.886535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.886609 (+ 74us) server_negotiation.cc:407] Connection header received
0810 07:25:19.886653 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.886653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.886666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.886724 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.886906 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34784: BlockingRecv error: recv got EOF from 127.0.0.1:34784 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:25:19.887046 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3138
I20260810 07:25:19.896826 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3293
I20260810 07:25:19.903512 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3443
I20260810 07:25:19.910024 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3030
2026-08-10T07:25:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (685 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T07:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:19Z Disabled control of system clock
I20260810 07:25:19.941058 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34141
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 07:25:20.026015 3613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:20.026245 3613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:20.026269 3613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:20.027444 3613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:20.027484 3613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:20.027498 3613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:20.027513 3613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:20.027526 3613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:20.028884 3613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:34141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:20.029102 3613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:20.029464 3613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:20.034482 3621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.034435 3620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.034441 3624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:20.034598 3613 server_base.cc:1034] running on GCE node
I20260810 07:25:20.035002 3613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:20.035276 3613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:20.036418 3613 hybrid_clock.cc:648] HybridClock initialized: now 1786346720036396 us; error 40 us; skew 500 ppm
I20260810 07:25:20.037798 3613 webserver.cc:460] Webserver started at http://127.24.60.254:38577/ using document root <none> and password file <none>
I20260810 07:25:20.038101 3613 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:20.038158 3613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:20.038394 3613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:20.039544 3613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "a8ef8556fc864d5a9b1a14e293f00f7a"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "ca60eab362964b425ddf5131f56ff255"
server_key_iv: "301ad11738cc7d58ce5797c2000436ae"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.039937 3613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "a8ef8556fc864d5a9b1a14e293f00f7a"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "ca60eab362964b425ddf5131f56ff255"
server_key_iv: "301ad11738cc7d58ce5797c2000436ae"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.041489 3613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:25:20.042400 3630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.042645 3613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:20.042763 3613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "a8ef8556fc864d5a9b1a14e293f00f7a"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "ca60eab362964b425ddf5131f56ff255"
server_key_iv: "301ad11738cc7d58ce5797c2000436ae"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.042847 3613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:20.046799 3613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:20.047128 3613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:20.047287 3613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:20.051613 3613 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:34141
I20260810 07:25:20.051647 3694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:34141 every 8 connection(s)
I20260810 07:25:20.052043 3613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:20.052785 3695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:20.055867 3695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a: Bootstrap starting.
I20260810 07:25:20.055886 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3613
I20260810 07:25:20.055991 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:25:20.056205 24819 external_mini_cluster.cc:1428] Setting key e04ac09948bc616877f57b1bdf45d87f
I20260810 07:25:20.056720 3695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:20.057128 3695 log.cc:824] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:20.057971 3695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a: No bootstrap required, opened a new log
I20260810 07:25:20.059284 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.056803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.24.60.254:34141)
0810 07:25:20.057060 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:20.057070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:20.057107 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:25:20.057734 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:20.057744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:20.057799 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:20.057971 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:20.058758 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:20.058763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:20.058763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:20.058769 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:20.058820 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:20.058836 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:20.058932 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:20.059014 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 07:25:20.059074 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260810 07:25:20.059901 3695 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34141 } }
I20260810 07:25:20.060101 3695 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:20.060150 3695 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ef8556fc864d5a9b1a14e293f00f7a, State: Initialized, Role: FOLLOWER
I20260810 07:25:20.060274 3695 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34141 } }
I20260810 07:25:20.060359 3695 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:20.060405 3695 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:20.060462 3695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:20.061441 3695 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34141 } }
I20260810 07:25:20.061580 3695 leader_election.cc:302] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ef8556fc864d5a9b1a14e293f00f7a; no voters:
I20260810 07:25:20.061774 3695 leader_election.cc:288] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:20.061832 3700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:20.062058 3700 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [term 1 LEADER]: Becoming Leader. State: Replica: a8ef8556fc864d5a9b1a14e293f00f7a, State: Running, Role: LEADER
I20260810 07:25:20.062069 3695 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:20.062211 3700 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34141 } }
I20260810 07:25:20.062806 3703 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ef8556fc864d5a9b1a14e293f00f7a. Latest consensus state: current_term: 1 leader_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34141 } } }
I20260810 07:25:20.062947 3703 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:20.062781 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ef8556fc864d5a9b1a14e293f00f7a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34141 } } }
I20260810 07:25:20.063059 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:20.063695 3712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:20.064224 3712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:20.066104 3712 catalog_manager.cc:1347] Generated new cluster ID: 771ee66693224009bf5613ada3cc0e2e
I20260810 07:25:20.066164 3712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:20.081703 3712 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:20.081928 3712 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:20.086478 3712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ef8556fc864d5a9b1a14e293f00f7a: Generated new TSK 0
I20260810 07:25:20.086719 3712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:20.090334 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34141
--builtin_ntp_servers=127.24.60.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 07:25:20.179908 3725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:20.180174 3725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:20.180217 3725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:20.181529 3725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:20.181600 3725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:20.181648 3725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:25:20.183068 3725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:20.183358 3725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:20.183682 3725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:20.189740 3734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.189766 3732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.189733 3731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:20.190047 3725 server_base.cc:1034] running on GCE node
I20260810 07:25:20.190251 3725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:20.190534 3725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:20.191701 3725 hybrid_clock.cc:648] HybridClock initialized: now 1786346720191677 us; error 41 us; skew 500 ppm
I20260810 07:25:20.193089 3725 webserver.cc:460] Webserver started at http://127.24.60.193:34205/ using document root <none> and password file <none>
I20260810 07:25:20.193398 3725 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:20.193447 3725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:20.193588 3725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:20.194730 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "a2da9c3b99f94a9080147002543b669e"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "58f3025c1dcab3b2fa3c7ff2ed958d13"
server_key_iv: "5d56206d98af47520b5c723972954b06"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.195338 3725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "a2da9c3b99f94a9080147002543b669e"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "58f3025c1dcab3b2fa3c7ff2ed958d13"
server_key_iv: "5d56206d98af47520b5c723972954b06"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.196911 3725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:25:20.197813 3740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.198016 3725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:20.198112 3725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "a2da9c3b99f94a9080147002543b669e"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "58f3025c1dcab3b2fa3c7ff2ed958d13"
server_key_iv: "5d56206d98af47520b5c723972954b06"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.198185 3725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:20.201643 3725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:20.201932 3725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:20.202090 3725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:20.202311 3725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:20.202633 3725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:20.202667 3725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.202719 3725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:20.202744 3725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.209353 3725 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46589
I20260810 07:25:20.209431 3870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46589 every 8 connection(s)
I20260810 07:25:20.209779 3725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:20.213083 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.210586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:34305 (local address 127.24.60.254:34141)
0810 07:25:20.210633 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:20.210636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:20.210889 (+ 253us) server_negotiation.cc:407] Connection header received
0810 07:25:20.211451 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:20.211452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:20.211460 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:20.211525 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:20.212766 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:20.212768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:20.212768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:20.212773 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:20.212790 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:20.212795 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:20.212877 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:20.212989 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 07:25:20.213009 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 07:25:20.213340 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.210546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34141 (local address 127.24.60.193:34305)
0810 07:25:20.210752 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:20.210788 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:20.211327 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:20.211614 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:20.211617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:20.211626 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:20.212594 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:20.212594 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:20.212627 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:20.212657 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:20.212874 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:20.212882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:20.212885 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:25:20.212994 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 07:25:20.213062 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260810 07:25:20.213879 3872 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34141
I20260810 07:25:20.213986 3872 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:20.214102 3872 heartbeater.cc:511] Master 127.24.60.254:34141 requested a full tablet report, sending...
I20260810 07:25:20.214629 3652 ts_manager.cc:194] Registered new tserver with Master: a2da9c3b99f94a9080147002543b669e (127.24.60.193:46589)
I20260810 07:25:20.216120 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3725
I20260810 07:25:20.216230 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:20.216418 24819 external_mini_cluster.cc:1428] Setting key 72d9287637e09998d01655d8c7bfa739
I20260810 07:25:20.217437 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34141
--builtin_ntp_servers=127.24.60.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 07:25:20.304726 3876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:20.304998 3876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:20.305042 3876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:20.306313 3876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:20.306391 3876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:20.306438 3876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:25:20.307852 3876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:20.308148 3876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:20.308492 3876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:20.314105 3882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:20.314214 3876 server_base.cc:1034] running on GCE node
W20260810 07:25:20.314105 3881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.314121 3885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:20.314643 3876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:20.314916 3876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:20.316123 3876 hybrid_clock.cc:648] HybridClock initialized: now 1786346720316100 us; error 30 us; skew 500 ppm
I20260810 07:25:20.317561 3876 webserver.cc:460] Webserver started at http://127.24.60.194:35585/ using document root <none> and password file <none>
I20260810 07:25:20.317826 3876 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:20.317900 3876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:20.318060 3876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:20.319221 3876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "3778bc65088d4997af4d057e5f184668"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "f5912401a9a250a0c599eb57a7a15282"
server_key_iv: "4c83d31fa15e44ff163e949f54094121"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.319608 3876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "3778bc65088d4997af4d057e5f184668"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "f5912401a9a250a0c599eb57a7a15282"
server_key_iv: "4c83d31fa15e44ff163e949f54094121"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.320940 3876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:25:20.321811 3891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.322029 3876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:20.322134 3876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "3778bc65088d4997af4d057e5f184668"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "f5912401a9a250a0c599eb57a7a15282"
server_key_iv: "4c83d31fa15e44ff163e949f54094121"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.322219 3876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:20.325850 3876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:20.326148 3876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:20.326296 3876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:20.326534 3876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:20.326884 3876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:20.326947 3876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.326992 3876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:20.327016 3876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.333703 3876 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:43091
I20260810 07:25:20.333776 4017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:43091 every 8 connection(s)
I20260810 07:25:20.334151 3876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:20.337529 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.334991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:37781 (local address 127.24.60.254:34141)
0810 07:25:20.335061 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:20.335063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:20.335313 (+ 250us) server_negotiation.cc:407] Connection header received
0810 07:25:20.335833 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:20.335834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:20.335841 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:20.335911 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:20.337269 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:20.337270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:20.337271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:20.337275 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:20.337292 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:20.337296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:20.337356 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:20.337451 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 07:25:20.337466 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:25:20.337731 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.334963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34141 (local address 127.24.60.194:37781)
0810 07:25:20.335176 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:20.335219 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:20.335712 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:20.335994 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:20.335998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:20.336007 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:20.337044 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:20.337044 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:20.337075 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:20.337106 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:20.337372 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:20.337378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:20.337381 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:25:20.337446 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:25:20.337501 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260810 07:25:20.338284 4018 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34141
I20260810 07:25:20.338375 4018 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:20.338562 4018 heartbeater.cc:511] Master 127.24.60.254:34141 requested a full tablet report, sending...
I20260810 07:25:20.338987 3652 ts_manager.cc:194] Registered new tserver with Master: 3778bc65088d4997af4d057e5f184668 (127.24.60.194:43091)
I20260810 07:25:20.343030 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 3876
I20260810 07:25:20.343175 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:20.343457 24819 external_mini_cluster.cc:1428] Setting key dfbb0e2b83887a8aefb3c17d8d8b78a8
I20260810 07:25:20.344704 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34141
--builtin_ntp_servers=127.24.60.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 07:25:20.451129 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:20.451366 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:20.451393 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:20.452648 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:20.452696 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:20.452723 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:25:20.454180 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:20.454466 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:20.454802 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:20.460130 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.460194 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:20.460121 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:20.460399 4021 server_base.cc:1034] running on GCE node
I20260810 07:25:20.460665 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:20.460924 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:20.462088 4021 hybrid_clock.cc:648] HybridClock initialized: now 1786346720462060 us; error 53 us; skew 500 ppm
I20260810 07:25:20.463542 4021 webserver.cc:460] Webserver started at http://127.24.60.195:46477/ using document root <none> and password file <none>
I20260810 07:25:20.463748 4021 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:20.463790 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:20.463886 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:20.464958 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "029407b4be8b4874a0a0e700f5df2001"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "1c59724f7650497965e355a00f3d4033"
server_key_iv: "8c069eb398eb42b77511de1da6de21a3"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.465387 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "029407b4be8b4874a0a0e700f5df2001"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "1c59724f7650497965e355a00f3d4033"
server_key_iv: "8c069eb398eb42b77511de1da6de21a3"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.466725 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:20.467447 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.467665 4021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:20.467762 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "029407b4be8b4874a0a0e700f5df2001"
format_stamp: "Formatted at 2026-08-10 07:25:20 on dist-test-slave-vxj2"
server_key: "1c59724f7650497965e355a00f3d4033"
server_key_iv: "8c069eb398eb42b77511de1da6de21a3"
server_key_version: "encryptionkey@0"
I20260810 07:25:20.467836 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:20.471387 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:20.471685 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:20.471834 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:20.472062 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:20.472397 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:20.472452 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.472502 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:20.472529 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:20.478735 4021 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:43453
I20260810 07:25:20.478816 4168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:43453 every 8 connection(s)
I20260810 07:25:20.479171 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:20.482115 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 4021
I20260810 07:25:20.482225 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:20.482352 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.479944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:52447 (local address 127.24.60.254:34141)
0810 07:25:20.480016 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:20.480018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:20.480285 (+ 267us) server_negotiation.cc:407] Connection header received
0810 07:25:20.480743 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:20.480744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:20.480750 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:20.480804 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:20.482088 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:20.482089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:20.482090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:20.482094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:20.482109 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:20.482113 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:20.482162 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:20.482263 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 07:25:20.482282 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:25:20.482450 24819 external_mini_cluster.cc:1428] Setting key 367358655c7a63534fc97f8a25176a19
I20260810 07:25:20.482560 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:20.479922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34141 (local address 127.24.60.195:52447)
0810 07:25:20.480114 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:20.480175 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:20.480654 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:20.480856 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:20.480859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:20.480866 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:20.481854 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:20.481855 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:20.481939 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:20.481981 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:20.482164 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:20.482169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:20.482172 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:25:20.482258 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:25:20.482330 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260810 07:25:20.483176 4169 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34141
I20260810 07:25:20.483275 4169 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:20.483428 4169 heartbeater.cc:511] Master 127.24.60.254:34141 requested a full tablet report, sending...
I20260810 07:25:20.483788 3650 ts_manager.cc:194] Registered new tserver with Master: 029407b4be8b4874a0a0e700f5df2001 (127.24.60.195:43453)
I20260810 07:25:20.494810 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 07:25:20.496860 3698 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:20.496401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.24.60.254:34141)
0810 07:25:20.496473 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:20.496475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:20.496485 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:25:20.496528 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:20.496529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:20.496536 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:20.496611 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:20.496777 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41308: BlockingRecv error: recv got EOF from 127.0.0.1:41308 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:25:20.496984 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3725
I20260810 07:25:20.504644 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3876
I20260810 07:25:20.512382 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 4021
I20260810 07:25:20.519021 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 3613
2026-08-10T07:25:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (608 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:20 dist-test-slave-vxj2 krb5kdc[4180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:20 dist-test-slave-vxj2 krb5kdc[4180](info): set up 2 sockets
Aug 10 07:25:20 dist-test-slave-vxj2 krb5kdc[4180](info): commencing operation
krb5kdc: starting...
W20260810 07:25:21.693298 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:25:21 dist-test-slave-vxj2 krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:25:21.827867 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45611
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:45611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:21.912848 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:21.913127 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:21.913172 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:21.914409 4194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:21.914497 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:21.914525 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:21.914546 4194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:21.914577 4194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:21.916108 4194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:45611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:45611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:21.916405 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:21.916656 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:21.921669 4201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.921698 4204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.921681 4200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:21.922051 4194 server_base.cc:1034] running on GCE node
I20260810 07:25:21.922335 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:21.922652 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:21.923818 4194 hybrid_clock.cc:648] HybridClock initialized: now 1786346721923800 us; error 28 us; skew 500 ppm
Aug 10 07:25:21 dist-test-slave-vxj2 krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:21.926652 4194 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:25:21.927165 4194 webserver.cc:460] Webserver started at http://127.24.60.254:37025/ using document root <none> and password file <none>
I20260810 07:25:21.927440 4194 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:21.927515 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:21.927671 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:21.928882 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "59381caa2a5c41b58ce161918df54cc8"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-vxj2"
server_key: "8f752dad73c2c9db895fd4c5171ca5ca"
server_key_iv: "17471066a48ce5a7f58718f0a17c77bf"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.929332 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "59381caa2a5c41b58ce161918df54cc8"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-vxj2"
server_key: "8f752dad73c2c9db895fd4c5171ca5ca"
server_key_iv: "17471066a48ce5a7f58718f0a17c77bf"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.930881 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:21.931674 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.931882 4194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:21.931998 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "59381caa2a5c41b58ce161918df54cc8"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-vxj2"
server_key: "8f752dad73c2c9db895fd4c5171ca5ca"
server_key_iv: "17471066a48ce5a7f58718f0a17c77bf"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.932082 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:21.942325 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:21.944814 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:21.944995 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:21.949028 4194 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:45611
I20260810 07:25:21.949064 4277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:45611 every 8 connection(s)
I20260810 07:25:21.949472 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:21.950536 4278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:21.952910 4278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8: Bootstrap starting.
I20260810 07:25:21.953312 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 4194
I20260810 07:25:21.953428 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:25:21.953619 24819 external_mini_cluster.cc:1428] Setting key a55f078759e8e3f1a375feef3d368fe0
I20260810 07:25:21.953665 4278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:21.953981 4278 log.cc:824] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:21.954797 4278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8: No bootstrap required, opened a new log
I20260810 07:25:21.957229 4278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59381caa2a5c41b58ce161918df54cc8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45611 } }
I20260810 07:25:21.957409 4278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:21.957442 4278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59381caa2a5c41b58ce161918df54cc8, State: Initialized, Role: FOLLOWER
I20260810 07:25:21.957568 4278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59381caa2a5c41b58ce161918df54cc8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45611 } }
I20260810 07:25:21.957648 4278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:21.957681 4278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:21.957729 4278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:21.958334 4278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59381caa2a5c41b58ce161918df54cc8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45611 } }
I20260810 07:25:21.958451 4278 leader_election.cc:302] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59381caa2a5c41b58ce161918df54cc8; no voters:
I20260810 07:25:21.958652 4278 leader_election.cc:288] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:21.958830 4284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:21.958945 4278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:21.959026 4284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [term 1 LEADER]: Becoming Leader. State: Replica: 59381caa2a5c41b58ce161918df54cc8, State: Running, Role: LEADER
I20260810 07:25:21.959139 4284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59381caa2a5c41b58ce161918df54cc8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45611 } }
I20260810 07:25:21.959468 4284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59381caa2a5c41b58ce161918df54cc8. Latest consensus state: current_term: 1 leader_uuid: "59381caa2a5c41b58ce161918df54cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59381caa2a5c41b58ce161918df54cc8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45611 } } }
I20260810 07:25:21.959570 4284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:21.959488 4283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59381caa2a5c41b58ce161918df54cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59381caa2a5c41b58ce161918df54cc8" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 45611 } } }
I20260810 07:25:21.959739 4283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:21.959889 4295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:21.960419 4295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 07:25:21 dist-test-slave-vxj2 krb5kdc[4180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:21.965262 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.954157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.24.60.254:45611)
0810 07:25:21.954352 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:21.954360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:21.954386 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:25:21.954821 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:21.954827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:21.954919 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:21.955043 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:21.955814 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.956663 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.957803 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.958239 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.958324 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.961577 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:21.961581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:21.961581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:21.961594 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:21.963591 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.963960 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.963963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.963963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.964088 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.964286 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.964287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.964288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.964616 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:21.964771 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:21.964846 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 07:25:21.965002 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260810 07:25:21.966293 4295 catalog_manager.cc:1347] Generated new cluster ID: ec28bb282beb4478ba1f4db587f0cf54
I20260810 07:25:21.966348 4295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:21.977069 4295 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:21.977833 4295 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:21.992158 4295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8: Generated new TSK 0
I20260810 07:25:21.992396 4295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:25:22.020401 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45611
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:22.106316 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:22.106617 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:22.106659 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:22.107913 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:22.107993 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:22.108040 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:25:22.109683 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:22.109982 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:22.110242 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:22.115720 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.115726 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.115950 4312 server_base.cc:1034] running on GCE node
W20260810 07:25:22.115999 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.116240 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:22.116514 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:22.117686 4312 hybrid_clock.cc:648] HybridClock initialized: now 1786346722117662 us; error 38 us; skew 500 ppm
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346722, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:22.125052 4312 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:25:22.125649 4312 webserver.cc:460] Webserver started at http://127.24.60.193:43379/ using document root <none> and password file <none>
I20260810 07:25:22.125941 4312 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:22.126016 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:22.126192 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:22.127341 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "f0a0dcf9743c4ef7b2f2b9398b240638"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "0c94f40292e42023b6452e9771d06d7b"
server_key_iv: "870bed53395a43578374936057b25d95"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.127733 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "f0a0dcf9743c4ef7b2f2b9398b240638"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "0c94f40292e42023b6452e9771d06d7b"
server_key_iv: "870bed53395a43578374936057b25d95"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.129145 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:25:22.129997 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.130201 4312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:22.130302 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "f0a0dcf9743c4ef7b2f2b9398b240638"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "0c94f40292e42023b6452e9771d06d7b"
server_key_iv: "870bed53395a43578374936057b25d95"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.130359 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:22.143126 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:22.145371 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:22.145510 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:22.145705 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:22.146039 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:22.146075 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.146097 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:22.146111 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.152148 4312 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:33423
I20260810 07:25:22.152236 4465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:33423 every 8 connection(s)
I20260810 07:25:22.152593 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:22.156452 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 4312
I20260810 07:25:22.156589 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:22.156788 24819 external_mini_cluster.cc:1428] Setting key 26bede28b8ce0a099c6f04bd5bfa4751
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346722, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:22.164013 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.153505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:56455 (local address 127.24.60.254:45611)
0810 07:25:22.153584 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.153586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.153804 (+ 218us) server_negotiation.cc:407] Connection header received
0810 07:25:22.154388 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.154389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.154419 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.154482 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.155521 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.156041 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.157554 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.157807 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.157891 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.160637 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.160639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.160639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:22.160648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.162432 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.162872 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.162873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.162874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.162946 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.163221 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.163222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.163223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.163397 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.163481 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.163750 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.163871 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:25:22.164022 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.153480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45611 (local address 127.24.60.193:56455)
0810 07:25:22.153678 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.153716 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.154274 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.154565 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.154573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.155194 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:22.155403 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.155408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.156157 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.156160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.157457 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.157465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.157900 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.157901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.158053 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.158644 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:22.158713 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:22.160538 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:22.162538 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:22.162546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:22.162548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:22.162783 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:22.163035 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:22.163036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:22.163037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:22.163147 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:22.163474 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:22.163480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:22.163605 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.163694 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.163824 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260810 07:25:22.164626 4467 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45611
I20260810 07:25:22.164727 4467 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:22.164968 4467 heartbeater.cc:511] Master 127.24.60.254:45611 requested a full tablet report, sending...
I20260810 07:25:22.165681 4233 ts_manager.cc:194] Registered new tserver with Master: f0a0dcf9743c4ef7b2f2b9398b240638 (127.24.60.193:33423)
I20260810 07:25:22.166692 4233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:56455
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:25:22.186481 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45611
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:22.272991 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:22.273252 4473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:22.273278 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:22.274506 4473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:22.274551 4473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:22.274577 4473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:25:22.276048 4473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:22.276306 4473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:22.276621 4473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:22.282753 4478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.282815 4479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.282753 4481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.282989 4473 server_base.cc:1034] running on GCE node
I20260810 07:25:22.283216 4473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:22.283463 4473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:22.284610 4473 hybrid_clock.cc:648] HybridClock initialized: now 1786346722284595 us; error 28 us; skew 500 ppm
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346722, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:22.287288 4473 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:25:22.287695 4473 webserver.cc:460] Webserver started at http://127.24.60.194:33149/ using document root <none> and password file <none>
I20260810 07:25:22.287884 4473 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:22.287930 4473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:22.288021 4473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:22.289059 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "18f88265c06d482e92c9811a7f6d19b0"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "e8653d47d34d952fe261959326eb28e3"
server_key_iv: "974424aae6b64e042cedcf31bbc1b101"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.289445 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "18f88265c06d482e92c9811a7f6d19b0"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "e8653d47d34d952fe261959326eb28e3"
server_key_iv: "974424aae6b64e042cedcf31bbc1b101"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.290795 4473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:22.291509 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.291680 4473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.291795 4473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "18f88265c06d482e92c9811a7f6d19b0"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "e8653d47d34d952fe261959326eb28e3"
server_key_iv: "974424aae6b64e042cedcf31bbc1b101"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.291886 4473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:22.304257 4473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:22.306413 4473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:22.306641 4473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:22.306907 4473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:22.307250 4473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:22.307310 4473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.307355 4473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:22.307384 4473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.313663 4473 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:38669
I20260810 07:25:22.313768 4619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:38669 every 8 connection(s)
I20260810 07:25:22.314105 4473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346722, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:22.322297 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 4473
I20260810 07:25:22.322431 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:22.322675 24819 external_mini_cluster.cc:1428] Setting key c24f176df967bf05c84bbfb90cc102c9
I20260810 07:25:22.323887 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.314935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:58597 (local address 127.24.60.254:45611)
0810 07:25:22.315023 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.315025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.315251 (+ 226us) server_negotiation.cc:407] Connection header received
0810 07:25:22.315761 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.315762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.315772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.315827 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.316827 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.317419 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.318537 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.318721 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.318771 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.320934 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.320935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.320936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:22.320943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.322274 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.322752 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.322754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.322754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.322825 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.323105 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.323106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.323106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.323225 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.323313 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.323666 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.323760 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:25:22.323901 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.314892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45611 (local address 127.24.60.194:58597)
0810 07:25:22.315118 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.315158 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.315670 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.315916 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.315919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.316513 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:22.316739 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.316744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.317503 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.317504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.318461 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.318464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.318793 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.318793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.318923 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.319409 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:22.319418 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:22.320848 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:22.322368 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:22.322376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:22.322378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:22.322670 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:22.322903 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:22.322904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:22.322904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:22.323039 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:22.323330 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:22.323339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:22.323521 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.323618 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.323742 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":112,"threads_started":1}
I20260810 07:25:22.324426 4620 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45611
I20260810 07:25:22.324549 4620 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:22.324736 4620 heartbeater.cc:511] Master 127.24.60.254:45611 requested a full tablet report, sending...
I20260810 07:25:22.325227 4233 ts_manager.cc:194] Registered new tserver with Master: 18f88265c06d482e92c9811a7f6d19b0 (127.24.60.194:38669)
I20260810 07:25:22.325616 4233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:58597
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:25:22.350530 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45611
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:22.435904 4625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:22.436128 4625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:22.436154 4625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:22.437374 4625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:22.437424 4625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:22.437453 4625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:25:22.438933 4625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:25:22.439198 4625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:22.439447 4625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:22.444782 4632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.444924 4625 server_base.cc:1034] running on GCE node
W20260810 07:25:22.444793 4631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.444759 4634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.445358 4625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:22.445587 4625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:22.446763 4625 hybrid_clock.cc:648] HybridClock initialized: now 1786346722446753 us; error 48 us; skew 500 ppm
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346722, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:22.449470 4625 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:25:22.449988 4625 webserver.cc:460] Webserver started at http://127.24.60.195:41589/ using document root <none> and password file <none>
I20260810 07:25:22.450246 4625 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:22.450317 4625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:22.450481 4625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:22.451603 4625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "a3d6efeb6de658bfe96d300384696af2"
server_key_iv: "d6db9e6e520de5328ba01d71541da143"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.451992 4625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "a3d6efeb6de658bfe96d300384696af2"
server_key_iv: "d6db9e6e520de5328ba01d71541da143"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.453434 4625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260810 07:25:22.454164 4641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.454334 4625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.454453 4625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-vxj2"
server_key: "a3d6efeb6de658bfe96d300384696af2"
server_key_iv: "d6db9e6e520de5328ba01d71541da143"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.454531 4625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:22.476564 4625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:22.478879 4625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:22.479053 4625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:22.479300 4625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:22.479641 4625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:22.479700 4625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.479748 4625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:22.479772 4625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.486020 4625 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:33425
I20260810 07:25:22.486068 4771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:33425 every 8 connection(s)
I20260810 07:25:22.486426 4625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346722, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:25:22.495882 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.487255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:45659 (local address 127.24.60.254:45611)
0810 07:25:22.487328 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.487330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.487545 (+ 215us) server_negotiation.cc:407] Connection header received
0810 07:25:22.488049 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.488051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.488061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.488124 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.489095 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.489613 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.490617 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.490804 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.490857 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.493069 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.493071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.493072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:22.493079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.494317 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.494811 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.494812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.494813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.494874 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.495137 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.495137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.495138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.495248 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.495303 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.495660 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.495748 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:25:22.496037 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.487220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:45611 (local address 127.24.60.195:45659)
0810 07:25:22.487406 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.487450 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.487959 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.488175 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.488179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.488776 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:22.488981 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.488987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.489697 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.489698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.490530 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.490533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.490891 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.490892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.490998 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.491477 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:22.491486 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:22.492962 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:22.494406 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:22.494415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:22.494422 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:22.494726 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:22.494957 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:22.494958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:22.494958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:22.495077 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:22.495332 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:22.495338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:22.495515 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.495637 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.495823 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260810 07:25:22.496368 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 4625
I20260810 07:25:22.496488 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:22.496563 4774 heartbeater.cc:347] Connected to a master server at 127.24.60.254:45611
I20260810 07:25:22.496691 4774 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:22.496697 24819 external_mini_cluster.cc:1428] Setting key 89fcc5c147cc7295c3471a29ae4340d8
I20260810 07:25:22.496883 4774 heartbeater.cc:511] Master 127.24.60.254:45611 requested a full tablet report, sending...
I20260810 07:25:22.497340 4233 ts_manager.cc:194] Registered new tserver with Master: ec0e308b9bed4a9bb8473a1c39be7c4d (127.24.60.195:33425)
I20260810 07:25:22.497750 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:25:22.497794 4233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:45659
I20260810 07:25:22.505246 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.499261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.24.60.254:45611)
0810 07:25:22.499327 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.499330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.499439 (+ 109us) server_negotiation.cc:407] Connection header received
0810 07:25:22.499517 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.499518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.499529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.499584 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.500157 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.500632 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.501423 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.501627 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.501703 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.502823 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.502824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.502825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:22.502832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.504025 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.504365 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.504367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.504367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.504428 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:22.504644 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:22.504644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:22.504645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:22.504796 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.504917 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.505019 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.505102 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:25:22.506970 4233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:22.508509 4233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:22.515125 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.511025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:38669 (local address 127.0.0.1:33920)
0810 07:25:22.511231 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.511240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.511279 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.511545 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.511548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.511551 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.511621 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.511624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.512713 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.512714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.513988 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.513991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.514790 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.514791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.514872 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.514886 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.514988 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.515046 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":66,"threads_started":1}
I20260810 07:25:22.515138 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.511155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.24.60.194:38669)
0810 07:25:22.511364 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.511369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.511378 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:25:22.511427 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.511429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.511441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.511526 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.511717 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.512605 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.514083 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.514682 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.514770 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.514835 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.515014 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.515060 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
I20260810 07:25:22.515404 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.510815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:33425 (local address 127.0.0.1:41748)
0810 07:25:22.511140 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.511184 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.511258 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.511889 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.511890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.511892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.511938 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.511939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.513219 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.513220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.514211 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.514214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.515117 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.515118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.515208 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.515212 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.515269 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.515323 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":16,"thread_start_us":99,"threads_started":1}
I20260810 07:25:22.515506 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.510812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:33423 (local address 127.0.0.1:38906)
0810 07:25:22.511139 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.511184 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.511259 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.511654 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.511655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.511657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.511722 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.511724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.513016 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.513017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.514128 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.514130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.515238 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.515239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.515314 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.515318 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.515370 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.515413 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":21,"thread_start_us":48,"threads_started":1}
I20260810 07:25:22.515518 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.510939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.24.60.193:33423)
0810 07:25:22.511179 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.511186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.511352 (+ 166us) server_negotiation.cc:407] Connection header received
0810 07:25:22.511396 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.511398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.511420 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.511509 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.511813 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.512901 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.514213 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.514946 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.515025 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.515080 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.515389 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.515436 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260810 07:25:22.515687 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.510925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.24.60.195:33425)
0810 07:25:22.511671 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.511675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.511685 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:25:22.511722 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.511724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.511738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.511812 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.512053 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.513090 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.514285 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.514966 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.515025 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.515080 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.515570 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.515613 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":102,"threads_started":1}
I20260810 07:25:22.516206 4541 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe4813a840e4f03aca115cbd7634410 (DEFAULT_TABLE table=test-table [id=6e6249b535ef4941ac75d00a3f537e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:22.516222 4699 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe4813a840e4f03aca115cbd7634410 (DEFAULT_TABLE table=test-table [id=6e6249b535ef4941ac75d00a3f537e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:22.516564 4699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe4813a840e4f03aca115cbd7634410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.516566 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe4813a840e4f03aca115cbd7634410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.520023 4793 tablet_bootstrap.cc:492] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Bootstrap starting.
I20260810 07:25:22.520023 4794 tablet_bootstrap.cc:492] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Bootstrap starting.
I20260810 07:25:22.520731 4794 tablet_bootstrap.cc:654] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.520730 4793 tablet_bootstrap.cc:654] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.520676 4382 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe4813a840e4f03aca115cbd7634410 (DEFAULT_TABLE table=test-table [id=6e6249b535ef4941ac75d00a3f537e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:22.521013 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe4813a840e4f03aca115cbd7634410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.521466 4794 log.cc:824] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.521466 4793 log.cc:824] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.522368 4794 tablet_bootstrap.cc:492] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: No bootstrap required, opened a new log
I20260810 07:25:22.522472 4794 ts_tablet_manager.cc:1397] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 07:25:22.522514 4793 tablet_bootstrap.cc:492] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: No bootstrap required, opened a new log
I20260810 07:25:22.522589 4793 ts_tablet_manager.cc:1397] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:22.523875 4794 raft_consensus.cc:348] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.524044 4794 raft_consensus.cc:374] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.524091 4794 raft_consensus.cc:729] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f88265c06d482e92c9811a7f6d19b0, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.524245 4794 consensus_queue.cc:260] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.524550 4794 ts_tablet_manager.cc:1428] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 07:25:22.524555 4798 tablet_bootstrap.cc:492] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: Bootstrap starting.
I20260810 07:25:22.524677 4620 heartbeater.cc:503] Master 127.24.60.254:45611 was elected leader, sending a full tablet report...
I20260810 07:25:22.524674 4793 raft_consensus.cc:348] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.524833 4793 raft_consensus.cc:374] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.524875 4793 raft_consensus.cc:729] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0e308b9bed4a9bb8473a1c39be7c4d, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.524982 4793 consensus_queue.cc:260] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.525288 4774 heartbeater.cc:503] Master 127.24.60.254:45611 was elected leader, sending a full tablet report...
I20260810 07:25:22.525295 4793 ts_tablet_manager.cc:1428] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 07:25:22.525796 4798 tablet_bootstrap.cc:654] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.526619 4798 log.cc:824] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.527302 4798 tablet_bootstrap.cc:492] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: No bootstrap required, opened a new log
I20260810 07:25:22.527359 4798 ts_tablet_manager.cc:1397] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:22.528820 4798 raft_consensus.cc:348] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.528944 4798 raft_consensus.cc:374] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.528965 4798 raft_consensus.cc:729] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a0dcf9743c4ef7b2f2b9398b240638, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.529115 4798 consensus_queue.cc:260] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.529429 4798 ts_tablet_manager.cc:1428] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:22.529464 4467 heartbeater.cc:503] Master 127.24.60.254:45611 was elected leader, sending a full tablet report...
I20260810 07:25:22.542546 4799 raft_consensus.cc:482] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:22.542675 4799 raft_consensus.cc:504] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.543049 4799 leader_election.cc:288] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec0e308b9bed4a9bb8473a1c39be7c4d (127.24.60.195:33425), f0a0dcf9743c4ef7b2f2b9398b240638 (127.24.60.193:33423)
I20260810 07:25:22.546074 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.543241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:33425 (local address 127.24.60.194:41135)
0810 07:25:22.543309 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.543322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.543380 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.543585 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.543586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.543588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.543664 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.543666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.544314 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.544315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.545099 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.545102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.545766 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.545767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.545876 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.545881 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.545936 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.545985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 07:25:22.546087 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.543372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:41135 (local address 127.24.60.195:33425)
0810 07:25:22.543425 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.543428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.543454 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:25:22.543487 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.543488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.543513 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.543562 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.543740 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.544218 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.545256 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.545665 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.545756 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.545766 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.545951 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.546004 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:25:22.546176 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.543366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:53495 (local address 127.24.60.193:33423)
0810 07:25:22.543428 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.543430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.543617 (+ 187us) server_negotiation.cc:407] Connection header received
0810 07:25:22.543681 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.543682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.543694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.543766 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.543948 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.544398 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.545382 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.545811 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.545885 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.545893 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.546064 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.546105 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:25:22.546459 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.543243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:33423 (local address 127.24.60.194:53495)
0810 07:25:22.543545 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.543553 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.543621 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.543769 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.543769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.543771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.543856 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.543858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.544487 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.544489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.545294 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.545297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.545900 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.545901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.545977 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.545981 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.546292 (+ 311us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.546351 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":90,"thread_start_us":150,"threads_started":1}
I20260810 07:25:22.546443 4721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe4813a840e4f03aca115cbd7634410" candidate_uuid: "18f88265c06d482e92c9811a7f6d19b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" is_pre_election: true
I20260810 07:25:22.546663 4721 raft_consensus.cc:2454] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18f88265c06d482e92c9811a7f6d19b0 in term 0.
I20260810 07:25:22.546943 4491 leader_election.cc:302] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f88265c06d482e92c9811a7f6d19b0, ec0e308b9bed4a9bb8473a1c39be7c4d; no voters:
I20260810 07:25:22.546916 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe4813a840e4f03aca115cbd7634410" candidate_uuid: "18f88265c06d482e92c9811a7f6d19b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" is_pre_election: true
I20260810 07:25:22.547081 4411 raft_consensus.cc:2454] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18f88265c06d482e92c9811a7f6d19b0 in term 0.
I20260810 07:25:22.547101 4799 raft_consensus.cc:2781] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:22.547149 4799 raft_consensus.cc:482] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:22.547185 4799 raft_consensus.cc:3037] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.547984 4799 raft_consensus.cc:504] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.548151 4799 leader_election.cc:288] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [CANDIDATE]: Term 1 election: Requested vote from peers ec0e308b9bed4a9bb8473a1c39be7c4d (127.24.60.195:33425), f0a0dcf9743c4ef7b2f2b9398b240638 (127.24.60.193:33423)
I20260810 07:25:22.548286 4721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe4813a840e4f03aca115cbd7634410" candidate_uuid: "18f88265c06d482e92c9811a7f6d19b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d"
I20260810 07:25:22.548328 4411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe4813a840e4f03aca115cbd7634410" candidate_uuid: "18f88265c06d482e92c9811a7f6d19b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638"
I20260810 07:25:22.548365 4721 raft_consensus.cc:3037] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.548427 4411 raft_consensus.cc:3037] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.549103 4721 raft_consensus.cc:2454] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18f88265c06d482e92c9811a7f6d19b0 in term 1.
I20260810 07:25:22.549266 4411 raft_consensus.cc:2454] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18f88265c06d482e92c9811a7f6d19b0 in term 1.
I20260810 07:25:22.549312 4491 leader_election.cc:302] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f88265c06d482e92c9811a7f6d19b0, ec0e308b9bed4a9bb8473a1c39be7c4d; no voters:
I20260810 07:25:22.549422 4799 raft_consensus.cc:2781] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:22.549636 4799 raft_consensus.cc:686] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 1 LEADER]: Becoming Leader. State: Replica: 18f88265c06d482e92c9811a7f6d19b0, State: Running, Role: LEADER
I20260810 07:25:22.549744 4799 consensus_queue.cc:237] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } }
I20260810 07:25:22.550637 4231 catalog_manager.cc:5495] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18f88265c06d482e92c9811a7f6d19b0 (127.24.60.194). New cstate: current_term: 1 leader_uuid: "18f88265c06d482e92c9811a7f6d19b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18f88265c06d482e92c9811a7f6d19b0" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 38669 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:22.560791 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.557961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.24.60.194:38669)
0810 07:25:22.558016 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.558019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.558030 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:25:22.558096 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.558097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.558121 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.558212 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:22.558393 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.558894 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.559684 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.559973 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.560046 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.560297 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:22.560516 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:22.560633 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.560653 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.560675 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 07:25:22.562935 4411 raft_consensus.cc:1261] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 1 FOLLOWER]: Refusing update from remote peer 18f88265c06d482e92c9811a7f6d19b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:22.562935 4721 raft_consensus.cc:1261] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 1 FOLLOWER]: Refusing update from remote peer 18f88265c06d482e92c9811a7f6d19b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:22.563305 4799 consensus_queue.cc:1035] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0e308b9bed4a9bb8473a1c39be7c4d" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:22.563467 4805 consensus_queue.cc:1035] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a0dcf9743c4ef7b2f2b9398b240638" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:25:22.564654 4621 tablet.cc:2369] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:22.566794 4809 mvcc.cc:205] Tried to move back new op lower bound from 7316876175614386176 to 7316876175564410880. Current Snapshot: MvccSnapshot[applied={T|T < 7316876175614386176 or (T in {7316876175614386176})}]
I20260810 07:25:22.568424 4233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:22.568545 4233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:22.569444 4233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59381caa2a5c41b58ce161918df54cc8: Sending DeleteTablet for 3 replicas of tablet 5fe4813a840e4f03aca115cbd7634410
I20260810 07:25:22.569802 4382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe4813a840e4f03aca115cbd7634410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:22 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:38906
I20260810 07:25:22.570082 4819 tablet_replica.cc:330] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638: stopping tablet replica
I20260810 07:25:22.570196 4819 raft_consensus.cc:2229] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:22.570096 4699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe4813a840e4f03aca115cbd7634410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:22 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:41748
I20260810 07:25:22.570304 4819 raft_consensus.cc:2258] T 5fe4813a840e4f03aca115cbd7634410 P f0a0dcf9743c4ef7b2f2b9398b240638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:22.570850 4541 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe4813a840e4f03aca115cbd7634410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:22 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:33920
I20260810 07:25:22.571090 4821 tablet_replica.cc:330] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: stopping tablet replica
I20260810 07:25:22.571115 4820 tablet_replica.cc:330] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: stopping tablet replica
I20260810 07:25:22.571192 4821 raft_consensus.cc:2229] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:22.571223 4820 raft_consensus.cc:2229] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:22.571269 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 4312
I20260810 07:25:22.571331 4821 raft_consensus.cc:2258] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:22.571322 4820 raft_consensus.cc:2258] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:22.573444 4821 ts_tablet_manager.cc:1905] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:22.574853 4821 ts_tablet_manager.cc:1918] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:22.574939 4821 log.cc:1196] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/5fe4813a840e4f03aca115cbd7634410
I20260810 07:25:22.575254 4821 ts_tablet_manager.cc:1939] T 5fe4813a840e4f03aca115cbd7634410 P 18f88265c06d482e92c9811a7f6d19b0: Deleting consensus metadata
I20260810 07:25:22.575279 4820 ts_tablet_manager.cc:1905] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:22.575771 4213 catalog_manager.cc:4841] TS 18f88265c06d482e92c9811a7f6d19b0 (127.24.60.194:38669): tablet 5fe4813a840e4f03aca115cbd7634410 (table test-table [id=6e6249b535ef4941ac75d00a3f537e84]) successfully deleted
I20260810 07:25:22.576851 4820 ts_tablet_manager.cc:1918] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:22.577013 4820 log.cc:1196] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/5fe4813a840e4f03aca115cbd7634410
I20260810 07:25:22.577374 4820 ts_tablet_manager.cc:1939] T 5fe4813a840e4f03aca115cbd7634410 P ec0e308b9bed4a9bb8473a1c39be7c4d: Deleting consensus metadata
I20260810 07:25:22.577823 4213 catalog_manager.cc:4841] TS ec0e308b9bed4a9bb8473a1c39be7c4d (127.24.60.195:33425): tablet 5fe4813a840e4f03aca115cbd7634410 (table test-table [id=6e6249b535ef4941ac75d00a3f537e84]) successfully deleted
W20260810 07:25:22.577868 4216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:33423 (error 108)
I20260810 07:25:22.578441 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.578281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:33423 (local address 127.0.0.1:38910)
0810 07:25:22.578343 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.578396 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:33423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 07:25:22.578533 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 4473
W20260810 07:25:22.578578 4216 catalog_manager.cc:4584] TS f0a0dcf9743c4ef7b2f2b9398b240638 (127.24.60.193:33423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fe4813a840e4f03aca115cbd7634410: Network error: Client connection negotiation failed: client connection to 127.24.60.193:33423: connect: Connection refused (error 111)
I20260810 07:25:22.586686 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 4625
I20260810 07:25:22.594754 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 4194
2026-08-10T07:25:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2076 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4825](info): set up 2 sockets
Aug 10 07:25:22 dist-test-slave-vxj2 krb5kdc[4825](info): commencing operation
krb5kdc: starting...
W20260810 07:25:23.716964 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:25:23 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/rangeradmin_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/rangeradmin_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/rangerlookup_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/rangerlookup_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
I20260810 07:25:23.894194 24819 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskghm2j2/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/postgres -l logfile start
2026-08-10 07:25:26.763 UTC [4860] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 07:25:26.763 UTC [4860] LOG: listening on IPv4 address "127.24.60.212", port 45381
2026-08-10 07:25:26.767 UTC [4860] LOG: listening on Unix socket "/tmp/.s.PGSQL.45381"
2026-08-10 07:25:26.792 UTC [4863] LOG: database system was shut down at 2026-08-10 07:25:25 UTC
2026-08-10 07:25:26.796 UTC [4860] LOG: database system is ready to accept connections
I20260810 07:25:27.858230 24819 mini_postgres.cc:96] Postgres bound to 45381
2026-08-10 07:25:27.865 UTC [4872] FATAL: database "slave" does not exist
127.24.60.212:45381 - accepting connections
I20260810 07:25:27.866430 24819 mini_ranger.cc:161] Starting Ranger...
I20260810 07:25:27.884231 24819 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 07:25:28.454495 24819 mini_ranger.cc:87] Created ranger Postgres database
I20260810 07:25:28.454573 24819 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-admin
2026-08-10 07:25:28,747 [I] DB FLAVOR :POSTGRES
2026-08-10 07:25:28,747 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:25:28,747 [I] Checking connection..
2026-08-10 07:25:28,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 07:25:29,003 [I] Checking connection passed.
2026-08-10 07:25:29,003 [I] --------- Verifying version history table ---------
2026-08-10 07:25:29,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:25:29,277 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:25:29,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:25:29,552 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:25:29,552 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 07:25:29,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 07:25:29.829 UTC [4949] WARNING: there is no transaction in progress
2026-08-10 07:25:29,837 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 07:25:29,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:25:30,116 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 07:25:30,117 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:25:30,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 07:25:30,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 07:25:30,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vxj2.c.gcp-upstream.internal','N') ;"
2026-08-10 07:25:30,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:25:31,236 [I] Table x_portal_user does not exist in database ranger
2026-08-10 07:25:31,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:25:31,525 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 07:25:31,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:25:31,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:25:32,057 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 07:25:32,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 07:25:33.205 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.224 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.244 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.262 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.282 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.304 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.374 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.384 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.405 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.418 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.430 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.447 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.459 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.467 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.476 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:33.485 UTC [5074] WARNING: there is no transaction in progress
2026-08-10 07:25:34,261 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 07:25:34,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 07:25:34,524 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 07:25:34,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:25:34,808 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 07:25:34,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:25:35,085 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 07:25:35,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:25:35,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:25:35,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vxj2.c.gcp-upstream.internal';"
2026-08-10 07:25:35,864 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 07:25:35,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 07:25:36,140 [I] DB_PATCHES have already been applied
I20260810 07:25:36.146497 24819 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-admin:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskghm2j2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 07:25:36.146597 24819 mini_ranger.cc:193] Using host: 127.24.60.212
I20260810 07:25:36.150286 24819 mini_ranger.cc:227] Ranger admin URL: http://127.24.60.212:54969
Aug 10, 2026 7:25:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:25:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:25:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:25:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:25:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:25:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:25:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 07:25:37.470131 24819 mini_ranger.cc:160] Time spent starting Ranger: real 9.604s user 0.001s sys 0.010s
Aug 10, 2026 7:25:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:25:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:25:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:25:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:26:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 07:26:01.205042 24819 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/rangerkms_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/rangerkms_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.60.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.60.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 07:26:01 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346761, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 07:26:01.304172 24819 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 07:26:01.315567 24819 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 07:26:02.404197 24819 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 07:26:02.404294 24819 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms
I20260810 07:26:02.404312 24819 mini_ranger_kms.cc:227] Using postgres at 127.24.60.212:45381
2026-08-10 07:26:02,621 [I] DB FLAVOR :POSTGRES
2026-08-10 07:26:02,622 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:26:02,622 [I] Checking connection
2026-08-10 07:26:02,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 07:26:02,929 [I] connection success
2026-08-10 07:26:02,929 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 07:26:02,929 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 07:26:02,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 07:26:03,200 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 07:26:03,200 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:26:03,200 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 07:26:03,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:45381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 07:26:03,514 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 07:26:04.087844 24819 mini_ranger_kms.cc:310] Created kms service
I20260810 07:26:04.291735 24819 mini_ranger_kms.cc:326] Created kudu user
I20260810 07:26:04.347152 24819 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 07:26:04 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346761, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:04.798918 24819 mini_ranger_kms.cc:384] Added ranger policy
I20260810 07:26:04.799218 24819 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms:/tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 07:26:04.799248 24819 mini_ranger_kms.cc:240] Using host: 127.24.60.212
I20260810 07:26:04.803524 24819 mini_ranger_kms.cc:276] Ranger KMS PID: 5298
I20260810 07:26:04.803696 24819 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.60.212:49123
Aug 10, 2026 7:26:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:26:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:26:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:26:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:26:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:26:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 07:26:07 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346767, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.60.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:07.419500 24819 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.115s user 0.007s sys 0.191s
I20260810 07:26:07.419629 24819 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 07:26:07.419709 24819 mini_ranger_kms.cc:401] 127.24.60.212:49123/kms/v1/keys
Aug 10 07:26:07 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346767, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.60.212@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
Aug 10 07:26:08 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346767, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.60.212@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
Aug 10 07:26:10 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:26:10.545666 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:41217
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:41217
--ranger_config_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:49123/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:10.637248 5358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:10.637501 5358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:10.637527 5358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:10.638797 5358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:26:10.638851 5358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:10.638865 5358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:10.638878 5358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:26:10.638891 5358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:26:10.640607 5358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.60.212:49123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:41217
--ranger_config_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:41217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:26:10.640838 5358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:10.641029 5358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:10.646888 5365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:10.646974 5358 server_base.cc:1034] running on GCE node
W20260810 07:26:10.646950 5363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:10.646950 5367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:10.647557 5358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:10.647868 5358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:10.649050 5358 hybrid_clock.cc:648] HybridClock initialized: now 1786346770649036 us; error 32 us; skew 500 ppm
Aug 10 07:26:10 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346770, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:10.657286 5358 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:26:10.657794 5358 webserver.cc:460] Webserver started at http://127.24.60.254:41751/ using document root <none> and password file <none>
I20260810 07:26:10.658066 5358 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:10.658133 5358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:10.658286 5358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:10 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346770, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:10.732287 5358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "e01a89cd82464cf1af4377e68c81811e"
format_stamp: "Formatted at 2026-08-10 07:26:10 on dist-test-slave-vxj2"
server_key: "195f30cde071797bbc0d7c819ae42078"
server_key_iv: "3ec9d28e5695d1e965a2ff5ceb76e846"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:10.732671 5358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "e01a89cd82464cf1af4377e68c81811e"
format_stamp: "Formatted at 2026-08-10 07:26:10 on dist-test-slave-vxj2"
server_key: "195f30cde071797bbc0d7c819ae42078"
server_key_iv: "3ec9d28e5695d1e965a2ff5ceb76e846"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:10.734372 5358 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 07:26:10.750484 5375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:10.750690 5358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:26:10.750805 5358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "e01a89cd82464cf1af4377e68c81811e"
format_stamp: "Formatted at 2026-08-10 07:26:10 on dist-test-slave-vxj2"
server_key: "195f30cde071797bbc0d7c819ae42078"
server_key_iv: "3ec9d28e5695d1e965a2ff5ceb76e846"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:10.750989 5358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:10.765905 5358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:10.768049 5358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:10.768218 5358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:10.773710 5358 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:41217
I20260810 07:26:10.773733 5439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:41217 every 8 connection(s)
I20260810 07:26:10.774233 5358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:26:10.774966 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:10.777596 5440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e: Bootstrap starting.
I20260810 07:26:10.778276 5440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:10.778596 5440 log.cc:824] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:10.779356 5440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e: No bootstrap required, opened a new log
I20260810 07:26:10.781630 5440 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01a89cd82464cf1af4377e68c81811e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41217 } }
I20260810 07:26:10.781880 5440 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:10.781952 5440 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01a89cd82464cf1af4377e68c81811e, State: Initialized, Role: FOLLOWER
I20260810 07:26:10.782146 5440 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01a89cd82464cf1af4377e68c81811e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41217 } }
I20260810 07:26:10.782254 5440 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:26:10.782300 5440 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:26:10.782351 5440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:10.782990 5440 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01a89cd82464cf1af4377e68c81811e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41217 } }
I20260810 07:26:10.783128 5440 leader_election.cc:302] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01a89cd82464cf1af4377e68c81811e; no voters:
I20260810 07:26:10.783332 5440 leader_election.cc:288] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:26:10.783416 5445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:26:10.783643 5445 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [term 1 LEADER]: Becoming Leader. State: Replica: e01a89cd82464cf1af4377e68c81811e, State: Running, Role: LEADER
I20260810 07:26:10.783648 5440 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:26:10.783756 5445 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01a89cd82464cf1af4377e68c81811e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41217 } }
I20260810 07:26:10.784118 5445 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01a89cd82464cf1af4377e68c81811e. Latest consensus state: current_term: 1 leader_uuid: "e01a89cd82464cf1af4377e68c81811e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01a89cd82464cf1af4377e68c81811e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41217 } } }
I20260810 07:26:10.784198 5445 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [sys.catalog]: This master's current role is: LEADER
I20260810 07:26:10.784132 5446 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01a89cd82464cf1af4377e68c81811e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01a89cd82464cf1af4377e68c81811e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 41217 } } }
I20260810 07:26:10.784238 5446 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e [sys.catalog]: This master's current role is: LEADER
I20260810 07:26:10.784380 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 5358
I20260810 07:26:10.784499 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:26:10.784617 5450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:26:10.784709 24819 external_mini_cluster.cc:1428] Setting key 33751ae7ca5b5351962756abb0ce0a52
I20260810 07:26:10.785545 5450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:26:10.786635 5440 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 07:26:10.793488 5450 catalog_manager.cc:1347] Generated new cluster ID: 2ff75f0a89b54faeafde9a584b43b20b
I20260810 07:26:10.793767 5450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:26:10 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:26:10.805912 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:10.785251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.24.60.254:41217)
0810 07:26:10.786265 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:10.786276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:10.786322 (+ 46us) server_negotiation.cc:407] Connection header received
0810 07:26:10.786557 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:10.786566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:10.786639 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:10.786849 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:10.787685 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:10.795755 (+ 8070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:10.797112 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:10.797906 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:10.798031 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:10.801312 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:10.801319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:10.801319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:10.801337 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:10.803608 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:10.804078 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:10.804081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:10.804082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:10.804194 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:10.804400 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:10.804401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:10.804401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:10.804788 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:10.804958 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:10.805509 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 07:26:10.805648 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":75,"threads_started":1}
I20260810 07:26:10.807591 5450 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:26:10.808319 5450 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:26:10.814960 5450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e: Generated new TSK 0
I20260810 07:26:10.815222 5450 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:26:11.537114 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:49123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41217
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:11.656797 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:11.657054 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:11.657081 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:11.658493 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:11.658557 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:11.658587 5492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:26:11.660310 5492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.60.212:49123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:26:11.660595 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:11.660871 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:11.666694 5501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:11.666697 5498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:11.666694 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:11.667218 5492 server_base.cc:1034] running on GCE node
I20260810 07:26:11.667500 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:11.667810 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:11.669023 5492 hybrid_clock.cc:648] HybridClock initialized: now 1786346771669000 us; error 41 us; skew 500 ppm
Aug 10 07:26:11 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:11.672060 5492 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:26:11.672672 5492 webserver.cc:460] Webserver started at http://127.24.60.193:39495/ using document root <none> and password file <none>
I20260810 07:26:11.672955 5492 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:11.673005 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:11.673187 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:11 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:11.697674 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "b900e6d0a5ab4cc7af77c563aea09920"
format_stamp: "Formatted at 2026-08-10 07:26:11 on dist-test-slave-vxj2"
server_key: "392a3ca6b891670b4a544ce1ff4b768f"
server_key_iv: "eaa8edf667936a092116fe52fe6dfe33"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:11.698243 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "b900e6d0a5ab4cc7af77c563aea09920"
format_stamp: "Formatted at 2026-08-10 07:26:11 on dist-test-slave-vxj2"
server_key: "392a3ca6b891670b4a544ce1ff4b768f"
server_key_iv: "eaa8edf667936a092116fe52fe6dfe33"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:11.702353 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260810 07:26:11.716945 5515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:11.717252 5492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:26:11.717406 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "b900e6d0a5ab4cc7af77c563aea09920"
format_stamp: "Formatted at 2026-08-10 07:26:11 on dist-test-slave-vxj2"
server_key: "392a3ca6b891670b4a544ce1ff4b768f"
server_key_iv: "eaa8edf667936a092116fe52fe6dfe33"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:11.717674 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:11.744356 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:11.747427 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:11.747653 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:11.748039 5492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:11.748549 5492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:11.748618 5492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:11.748658 5492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:11.748677 5492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:11.757236 5492 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:44669
I20260810 07:26:11.757319 5657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:44669 every 8 connection(s)
I20260810 07:26:11.757774 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:26:11 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:26:11.766132 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 5492
I20260810 07:26:11.766258 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:26:11.766476 24819 external_mini_cluster.cc:1428] Setting key 1300168c92bb4d21607e66cbd5615ca5
I20260810 07:26:11.768644 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:11.758637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:33067 (local address 127.24.60.254:41217)
0810 07:26:11.758806 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:11.758808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:11.759050 (+ 242us) server_negotiation.cc:407] Connection header received
0810 07:26:11.759615 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:11.759616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:11.759634 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:11.759705 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:11.760327 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:11.760854 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:11.762020 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:11.762239 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:11.762338 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:11.764618 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:11.764619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:11.764619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:11.764627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:11.766567 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:11.767160 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:11.767162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:11.767162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:11.767237 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:11.767661 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:11.767662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:11.767663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:11.767831 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:11.767913 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:11.768358 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 07:26:11.768479 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260810 07:26:11.768847 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:11.758643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41217 (local address 127.24.60.193:33067)
0810 07:26:11.758916 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:11.758957 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:11.759495 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:11.759787 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:11.759792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:11.760009 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:11.760222 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:11.760227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:11.760989 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:11.760990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:11.761885 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:11.761888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:11.762353 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:11.762354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:11.762489 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:11.763090 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:11.763100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:11.764482 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:11.766685 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:11.766698 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:11.766701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:11.767066 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:11.767348 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:11.767349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:11.767350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:11.767555 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:11.767933 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:11.767943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:11.768163 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 07:26:11.768326 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 07:26:11.768562 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
I20260810 07:26:11.769697 5658 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41217
I20260810 07:26:11.769826 5658 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:11.770052 5658 heartbeater.cc:511] Master 127.24.60.254:41217 requested a full tablet report, sending...
I20260810 07:26:11.770802 5394 ts_manager.cc:194] Registered new tserver with Master: b900e6d0a5ab4cc7af77c563aea09920 (127.24.60.193:44669)
I20260810 07:26:11.771847 5394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:33067
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:26:11.804669 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:49123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41217
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:11.912405 5666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:11.912667 5666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:11.912711 5666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:11.914225 5666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:11.914341 5666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:11.914419 5666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:26:11.916280 5666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.60.212:49123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:26:11.916582 5666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:11.916831 5666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:11.922286 5679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:11.922412 5675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:11.922749 5676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:11.922956 5666 server_base.cc:1034] running on GCE node
I20260810 07:26:11.923256 5666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:11.923508 5666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Aug 10 07:26:11 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:11.924700 5666 hybrid_clock.cc:648] HybridClock initialized: now 1786346771924678 us; error 38 us; skew 500 ppm
Aug 10 07:26:11 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:11.927497 5666 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:26:11.928074 5666 webserver.cc:460] Webserver started at http://127.24.60.194:37881/ using document root <none> and password file <none>
I20260810 07:26:11.928349 5666 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:11.928418 5666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:11.928570 5666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:11 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:11.950757 5666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "de067db565a543c68ecc1ac0dfe8f464"
format_stamp: "Formatted at 2026-08-10 07:26:11 on dist-test-slave-vxj2"
server_key: "3137bf795a1ae927fe70d83165785f6e"
server_key_iv: "10dc39f6031f32fe80ac32a47d4252a6"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:11.951296 5666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "de067db565a543c68ecc1ac0dfe8f464"
format_stamp: "Formatted at 2026-08-10 07:26:11 on dist-test-slave-vxj2"
server_key: "3137bf795a1ae927fe70d83165785f6e"
server_key_iv: "10dc39f6031f32fe80ac32a47d4252a6"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:11.955174 5666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260810 07:26:11.978504 5689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:11.978729 5666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:26:11.978842 5666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "de067db565a543c68ecc1ac0dfe8f464"
format_stamp: "Formatted at 2026-08-10 07:26:11 on dist-test-slave-vxj2"
server_key: "3137bf795a1ae927fe70d83165785f6e"
server_key_iv: "10dc39f6031f32fe80ac32a47d4252a6"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:11.978948 5666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:11.998083 5666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:12.000898 5666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:12.001082 5666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:12.001412 5666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:12.001885 5666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:12.001945 5666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:12.001991 5666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:12.002017 5666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:12.009284 5666 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:46015
I20260810 07:26:12.009763 5666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:26:12.010648 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:46015 every 8 connection(s)
I20260810 07:26:12.013429 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 5666
I20260810 07:26:12.013546 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:26:12.013723 24819 external_mini_cluster.cc:1428] Setting key 1b1d95537030c30dd45af21b4f527544
Aug 10 07:26:12 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:26:12.022918 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.011420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41217 (local address 127.24.60.194:44089)
0810 07:26:12.011595 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.011646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.012323 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.012632 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.012637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.012907 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:12.013255 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.013261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.014250 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.014251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.015598 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.015602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.016206 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.016207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.016358 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.017049 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:12.017060 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:12.018865 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:12.021005 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:12.021014 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:12.021017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:12.021411 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:12.021645 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:12.021645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:12.021646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:12.021813 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:12.022168 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:12.022176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:12.022369 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.022476 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.022655 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260810 07:26:12.023722 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.011426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:44089 (local address 127.24.60.254:41217)
0810 07:26:12.011480 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.011482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.011762 (+ 280us) server_negotiation.cc:407] Connection header received
0810 07:26:12.012456 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.012458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.012476 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.012548 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:12.013376 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.014132 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.015715 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.016050 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.016100 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.018988 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:12.018990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:12.018991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:12.019001 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:12.020891 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:12.021508 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:12.021509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:12.021510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:12.021576 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:12.021916 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:12.021917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:12.021917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:12.022078 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:12.022138 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.023523 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.023628 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:26:12.024188 5821 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41217
I20260810 07:26:12.024304 5821 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:12.024523 5821 heartbeater.cc:511] Master 127.24.60.254:41217 requested a full tablet report, sending...
I20260810 07:26:12.024984 5394 ts_manager.cc:194] Registered new tserver with Master: de067db565a543c68ecc1ac0dfe8f464 (127.24.60.194:46015)
I20260810 07:26:12.025501 5394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:44089
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:26:12.052735 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:49123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41217
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:12.183300 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:12.183646 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:12.183707 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:12.185773 5826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:12.185859 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:12.185911 5826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:26:12.188520 5826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.60.212:49123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:41217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:26:12.188835 5826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:12.189234 5826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:12.195732 5835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:12.196171 5833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:12.196942 5837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:12.197722 5826 server_base.cc:1034] running on GCE node
I20260810 07:26:12.198004 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:12.198318 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:12.201614 5826 hybrid_clock.cc:648] HybridClock initialized: now 1786346772201604 us; error 31 us; skew 500 ppm
Aug 10 07:26:12 dist-test-slave-vxj2 krb5kdc[4825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346772, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:12.205341 5826 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:26:12.209322 5826 webserver.cc:460] Webserver started at http://127.24.60.195:36753/ using document root <none> and password file <none>
I20260810 07:26:12.209642 5826 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:12.209721 5826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:12.209923 5826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:12 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346772, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:12.224315 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "f1f85a7cd8b64e4488678496aea176ed"
format_stamp: "Formatted at 2026-08-10 07:26:12 on dist-test-slave-vxj2"
server_key: "16b4082c5e99edda209b574ca4d52599"
server_key_iv: "d4afbae73b9161cced014877926b0d02"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:12.224802 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "f1f85a7cd8b64e4488678496aea176ed"
format_stamp: "Formatted at 2026-08-10 07:26:12 on dist-test-slave-vxj2"
server_key: "16b4082c5e99edda209b574ca4d52599"
server_key_iv: "d4afbae73b9161cced014877926b0d02"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:12.227087 5826 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:26:12.242231 5845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:12.242511 5826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:26:12.242600 5826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "f1f85a7cd8b64e4488678496aea176ed"
format_stamp: "Formatted at 2026-08-10 07:26:12 on dist-test-slave-vxj2"
server_key: "16b4082c5e99edda209b574ca4d52599"
server_key_iv: "d4afbae73b9161cced014877926b0d02"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:12.242691 5826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:12.263499 5826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:12.266407 5826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:12.266685 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:12.266978 5826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:12.267443 5826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:12.267494 5826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:12.267526 5826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:12.267553 5826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:12.275561 5826 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:35821
I20260810 07:26:12.276060 5826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:26:12.278153 5976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:35821 every 8 connection(s)
I20260810 07:26:12.284149 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 5826
I20260810 07:26:12.284277 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:26:12.284510 24819 external_mini_cluster.cc:1428] Setting key 3c9e220674b3c7f00ab17d668eff0fb3
Aug 10 07:26:12 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346772, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:26:12.291570 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.276980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:59705 (local address 127.24.60.254:41217)
0810 07:26:12.277039 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.277042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.277773 (+ 731us) server_negotiation.cc:407] Connection header received
0810 07:26:12.278591 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.278593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.278609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.278648 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:12.279677 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.280568 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.284144 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.284482 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.284529 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.287761 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:12.287763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:12.287763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:12.287771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:12.289652 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:12.290327 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:12.290329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:12.290329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:12.290422 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:12.290757 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:12.290758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:12.290759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:12.290910 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:12.290977 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.291373 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.291469 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:26:12.292137 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.277058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:41217 (local address 127.24.60.195:59705)
0810 07:26:12.277600 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.277659 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.278443 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.278867 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.278873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.279182 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:12.279538 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.279547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.280706 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.280707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.284008 (+ 3301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.284013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.284673 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.284674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.284839 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.285647 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:12.285658 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:12.287640 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:12.289839 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:12.289847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:12.289851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:12.290184 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:12.290502 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:12.290503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:12.290504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:12.290657 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:12.291013 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:12.291020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:12.291201 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.291690 (+ 489us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.291881 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":112,"threads_started":1}
I20260810 07:26:12.292680 5977 heartbeater.cc:347] Connected to a master server at 127.24.60.254:41217
I20260810 07:26:12.292789 5977 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:12.293025 5977 heartbeater.cc:511] Master 127.24.60.254:41217 requested a full tablet report, sending...
I20260810 07:26:12.293440 5394 ts_manager.cc:194] Registered new tserver with Master: f1f85a7cd8b64e4488678496aea176ed (127.24.60.195:35821)
I20260810 07:26:12.293967 5394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:59705
I20260810 07:26:12.297274 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:26:12.308297 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.298946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.24.60.254:41217)
0810 07:26:12.298994 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.298996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.299173 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:26:12.299238 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.299239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.299251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.299320 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:12.300193 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.301018 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.302429 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.302771 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.302824 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.304819 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:12.304821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:12.304821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:12.304830 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:12.306830 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:12.307300 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:12.307301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:12.307302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:12.307380 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:12.307649 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:12.307650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:12.307651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:12.307809 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:12.307924 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.308106 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.308198 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:26:12.310750 5394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:26:12.313253 5394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:26:12.324218 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.317491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:46015 (local address 127.0.0.1:45794)
0810 07:26:12.317991 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.318040 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.318122 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.318950 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.318954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.318956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:12.319041 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.319044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.320873 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.320875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.322341 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.322345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.323879 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.323881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.323983 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.324002 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.324060 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.324118 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":89,"threads_started":1}
I20260810 07:26:12.329421 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.317798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:44669 (local address 127.0.0.1:58196)
0810 07:26:12.324430 (+ 6632us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.324442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.324489 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.324791 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.324793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.324794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:12.324828 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.324829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.326523 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.326525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.327602 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.327605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.328753 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.328754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.328816 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.328819 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.328874 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.329336 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6618,"thread_start_us":89,"threads_started":1}
I20260810 07:26:12.329994 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.317530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.24.60.194:46015)
0810 07:26:12.318742 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.318748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.318759 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:26:12.318801 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.318803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.318820 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.319131 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:12.319214 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.320773 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.322481 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.323765 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.329664 (+ 5899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.329797 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.329834 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.329898 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":94,"threads_started":1}
I20260810 07:26:12.330387 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.318004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:35821 (local address 127.0.0.1:57468)
0810 07:26:12.325261 (+ 7257us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.325275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.325354 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.325579 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.325580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.325582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:12.325629 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.325630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.327173 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.327174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.328374 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.328378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.329906 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.329908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.329986 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.329991 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.330026 (+ 35us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.330280 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7234,"thread_start_us":67,"threads_started":1}
I20260810 07:26:12.330768 5746 tablet_service.cc:1467] Processing CreateTablet for tablet f56c7274fe9e4d94b0413579737b5ef7 (DEFAULT_TABLE table=test-table [id=6988985795b1414ca2ff2fa1c5979e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:12.331107 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.318268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.24.60.195:35821)
0810 07:26:12.318473 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.318480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.325356 (+ 6876us) server_negotiation.cc:407] Connection header received
0810 07:26:12.325440 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.325443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.325463 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.325715 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:12.325805 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.327033 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.328575 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.329750 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.330716 (+ 966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.330854 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.330889 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.330939 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260810 07:26:12.331221 5746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56c7274fe9e4d94b0413579737b5ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:12.335085 5996 tablet_bootstrap.cc:492] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: Bootstrap starting.
I20260810 07:26:12.336148 5900 tablet_service.cc:1467] Processing CreateTablet for tablet f56c7274fe9e4d94b0413579737b5ef7 (DEFAULT_TABLE table=test-table [id=6988985795b1414ca2ff2fa1c5979e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:12.336549 5900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56c7274fe9e4d94b0413579737b5ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:12.336567 5996 tablet_bootstrap.cc:654] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:12.337075 5996 log.cc:824] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:12.337914 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.317350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.24.60.193:44669)
0810 07:26:12.320634 (+ 3284us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.320640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.324561 (+ 3921us) server_negotiation.cc:407] Connection header received
0810 07:26:12.324607 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.324611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.324630 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.324710 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:12.324907 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.326399 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.327710 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.328570 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.328646 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.328714 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.337727 (+ 9013us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.337808 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3243,"thread_start_us":76,"threads_started":1}
I20260810 07:26:12.338881 5996 tablet_bootstrap.cc:492] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: No bootstrap required, opened a new log
I20260810 07:26:12.338961 5996 ts_tablet_manager.cc:1397] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 07:26:12.340811 5574 tablet_service.cc:1467] Processing CreateTablet for tablet f56c7274fe9e4d94b0413579737b5ef7 (DEFAULT_TABLE table=test-table [id=6988985795b1414ca2ff2fa1c5979e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:12.341400 5996 raft_consensus.cc:348] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.341624 5996 raft_consensus.cc:374] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:12.341670 5996 raft_consensus.cc:729] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de067db565a543c68ecc1ac0dfe8f464, State: Initialized, Role: FOLLOWER
I20260810 07:26:12.341800 5996 consensus_queue.cc:260] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.342154 5996 ts_tablet_manager.cc:1428] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:26:12.342355 5821 heartbeater.cc:503] Master 127.24.60.254:41217 was elected leader, sending a full tablet report...
I20260810 07:26:12.342919 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56c7274fe9e4d94b0413579737b5ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:12.345597 5998 tablet_bootstrap.cc:492] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Bootstrap starting.
I20260810 07:26:12.347553 6001 tablet_bootstrap.cc:492] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920: Bootstrap starting.
I20260810 07:26:12.347993 5998 tablet_bootstrap.cc:654] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:12.348485 6001 tablet_bootstrap.cc:654] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:12.348848 6001 log.cc:824] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:12.348889 5998 log.cc:824] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:12.349841 5998 tablet_bootstrap.cc:492] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: No bootstrap required, opened a new log
I20260810 07:26:12.349840 6001 tablet_bootstrap.cc:492] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920: No bootstrap required, opened a new log
I20260810 07:26:12.349922 6001 ts_tablet_manager.cc:1397] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:26:12.349936 5998 ts_tablet_manager.cc:1397] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260810 07:26:12.352061 5998 raft_consensus.cc:348] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.352257 5998 raft_consensus.cc:374] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:12.352293 5998 raft_consensus.cc:729] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f85a7cd8b64e4488678496aea176ed, State: Initialized, Role: FOLLOWER
I20260810 07:26:12.352430 5998 consensus_queue.cc:260] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.352738 5998 ts_tablet_manager.cc:1428] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 07:26:12.353078 5977 heartbeater.cc:503] Master 127.24.60.254:41217 was elected leader, sending a full tablet report...
I20260810 07:26:12.353035 6001 raft_consensus.cc:348] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.353240 6001 raft_consensus.cc:374] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:12.353274 6001 raft_consensus.cc:729] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b900e6d0a5ab4cc7af77c563aea09920, State: Initialized, Role: FOLLOWER
I20260810 07:26:12.353480 6001 consensus_queue.cc:260] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.353775 6001 ts_tablet_manager.cc:1428] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 07:26:12.353907 5658 heartbeater.cc:503] Master 127.24.60.254:41217 was elected leader, sending a full tablet report...
I20260810 07:26:12.391811 6006 raft_consensus.cc:482] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:26:12.391996 6006 raft_consensus.cc:504] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.392524 6006 leader_election.cc:288] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f85a7cd8b64e4488678496aea176ed (127.24.60.195:35821), de067db565a543c68ecc1ac0dfe8f464 (127.24.60.194:46015)
I20260810 07:26:12.397511 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.392992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:48779 (local address 127.24.60.194:46015)
0810 07:26:12.393040 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.393042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.393630 (+ 588us) server_negotiation.cc:407] Connection header received
0810 07:26:12.393730 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.393731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.393743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.393774 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:12.394002 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.394765 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.396480 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.397039 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.397090 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.397099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.397385 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.397429 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:26:12.397698 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.392918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:45009 (local address 127.24.60.195:35821)
0810 07:26:12.393009 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.393012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.393496 (+ 484us) server_negotiation.cc:407] Connection header received
0810 07:26:12.393586 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.393587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.393602 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.393666 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:12.393967 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.394848 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.396519 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.397124 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.397217 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.397228 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.397537 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.397610 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:26:12.398069 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.392742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:35821 (local address 127.24.60.193:45009)
0810 07:26:12.393390 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.393404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.393497 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.393733 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.393735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.393737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:12.393859 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.393866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.394992 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.394992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.396387 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.396391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.397346 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.397347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.397438 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.397441 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.397919 (+ 478us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.397973 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":461,"threads_started":1}
I20260810 07:26:12.398434 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.392771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:46015 (local address 127.24.60.193:48779)
0810 07:26:12.393578 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.393587 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:12.393656 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:12.393843 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:12.393843 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:12.393845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:12.393930 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.393932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.394905 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.394906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.396351 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:12.396356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.397205 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.397206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:12.397287 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.397290 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:26:12.398055 (+ 765us) client_negotiation.cc:240] Negotiation successful
0810 07:26:12.398105 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":65,"thread_start_us":497,"threads_started":1}
I20260810 07:26:12.398487 5921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56c7274fe9e4d94b0413579737b5ef7" candidate_uuid: "b900e6d0a5ab4cc7af77c563aea09920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f85a7cd8b64e4488678496aea176ed" is_pre_election: true
I20260810 07:26:12.398686 5921 raft_consensus.cc:2454] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b900e6d0a5ab4cc7af77c563aea09920 in term 0.
I20260810 07:26:12.398929 5768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56c7274fe9e4d94b0413579737b5ef7" candidate_uuid: "b900e6d0a5ab4cc7af77c563aea09920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de067db565a543c68ecc1ac0dfe8f464" is_pre_election: true
I20260810 07:26:12.398988 5518 leader_election.cc:302] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b900e6d0a5ab4cc7af77c563aea09920, f1f85a7cd8b64e4488678496aea176ed; no voters:
I20260810 07:26:12.399070 5768 raft_consensus.cc:2454] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b900e6d0a5ab4cc7af77c563aea09920 in term 0.
I20260810 07:26:12.399188 6006 raft_consensus.cc:2781] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:26:12.399245 6006 raft_consensus.cc:482] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:26:12.399271 6006 raft_consensus.cc:3037] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:12.400506 6006 raft_consensus.cc:504] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.400662 6006 leader_election.cc:288] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [CANDIDATE]: Term 1 election: Requested vote from peers f1f85a7cd8b64e4488678496aea176ed (127.24.60.195:35821), de067db565a543c68ecc1ac0dfe8f464 (127.24.60.194:46015)
I20260810 07:26:12.400877 5768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56c7274fe9e4d94b0413579737b5ef7" candidate_uuid: "b900e6d0a5ab4cc7af77c563aea09920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de067db565a543c68ecc1ac0dfe8f464"
I20260810 07:26:12.400880 5921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56c7274fe9e4d94b0413579737b5ef7" candidate_uuid: "b900e6d0a5ab4cc7af77c563aea09920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f85a7cd8b64e4488678496aea176ed"
I20260810 07:26:12.400967 5768 raft_consensus.cc:3037] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:12.400983 5921 raft_consensus.cc:3037] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:12.402004 5768 raft_consensus.cc:2454] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b900e6d0a5ab4cc7af77c563aea09920 in term 1.
I20260810 07:26:12.402170 5921 raft_consensus.cc:2454] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b900e6d0a5ab4cc7af77c563aea09920 in term 1.
I20260810 07:26:12.402257 5517 leader_election.cc:302] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b900e6d0a5ab4cc7af77c563aea09920, de067db565a543c68ecc1ac0dfe8f464; no voters:
I20260810 07:26:12.402403 6006 raft_consensus.cc:2781] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:26:12.402575 6006 raft_consensus.cc:686] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [term 1 LEADER]: Becoming Leader. State: Replica: b900e6d0a5ab4cc7af77c563aea09920, State: Running, Role: LEADER
I20260810 07:26:12.402678 6006 consensus_queue.cc:237] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } }
I20260810 07:26:12.403550 5394 catalog_manager.cc:5495] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 reported cstate change: term changed from 0 to 1, leader changed from <none> to b900e6d0a5ab4cc7af77c563aea09920 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "b900e6d0a5ab4cc7af77c563aea09920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b900e6d0a5ab4cc7af77c563aea09920" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 44669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 } health_report { overall_health: UNKNOWN } } }
I20260810 07:26:12.430542 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.426547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.24.60.193:44669)
0810 07:26:12.426601 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:12.426603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:12.426613 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:26:12.426660 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:12.426661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:12.426689 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:12.426775 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:26:12.427025 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.427823 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.429140 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:12.429463 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:12.429547 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:12.429812 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:26:12.429964 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:26:12.430069 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:12.430404 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 07:26:12.430432 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 07:26:12.434536 5768 raft_consensus.cc:1261] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 1 FOLLOWER]: Refusing update from remote peer b900e6d0a5ab4cc7af77c563aea09920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:26:12.434773 5921 raft_consensus.cc:1261] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 1 FOLLOWER]: Refusing update from remote peer b900e6d0a5ab4cc7af77c563aea09920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:26:12.434909 6009 consensus_queue.cc:1035] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de067db565a543c68ecc1ac0dfe8f464" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:26:12.435491 6009 consensus_queue.cc:1035] T f56c7274fe9e4d94b0413579737b5ef7 P b900e6d0a5ab4cc7af77c563aea09920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f85a7cd8b64e4488678496aea176ed" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 35821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:26:12.439602 6012 mvcc.cc:205] Tried to move back new op lower bound from 7316876379887702016 to 7316876379761938432. Current Snapshot: MvccSnapshot[applied={T|T < 7316876379887702016}]
I20260810 07:26:12.439813 6014 mvcc.cc:205] Tried to move back new op lower bound from 7316876379887702016 to 7316876379761938432. Current Snapshot: MvccSnapshot[applied={T|T < 7316876379887702016}]
I20260810 07:26:12.441305 6013 mvcc.cc:205] Tried to move back new op lower bound from 7316876379887702016 to 7316876379761938432. Current Snapshot: MvccSnapshot[applied={T|T < 7316876379887702016}]
I20260810 07:26:12.444100 5394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:26:12.444227 5394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39434:
table { table_name: "test-table" } modify_external_catalogs: true
Aug 10 07:26:12 dist-test-slave-vxj2 krb5kdc[4825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346771, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:12.445099 5394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e01a89cd82464cf1af4377e68c81811e: Sending DeleteTablet for 3 replicas of tablet f56c7274fe9e4d94b0413579737b5ef7
I20260810 07:26:12.445534 5574 tablet_service.cc:1514] Processing DeleteTablet for tablet f56c7274fe9e4d94b0413579737b5ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:12 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:58196
I20260810 07:26:12.445734 5900 tablet_service.cc:1514] Processing DeleteTablet for tablet f56c7274fe9e4d94b0413579737b5ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:12 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:57468
I20260810 07:26:12.445995 6022 tablet_replica.cc:330] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: stopping tablet replica
I20260810 07:26:12.446079 5746 tablet_service.cc:1514] Processing DeleteTablet for tablet f56c7274fe9e4d94b0413579737b5ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:12 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:45794
I20260810 07:26:12.446271 6022 raft_consensus.cc:2229] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:26:12.446447 6022 raft_consensus.cc:2258] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:26:12.446555 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 5492
I20260810 07:26:12.448006 6022 ts_tablet_manager.cc:1905] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:26:12.450730 6022 ts_tablet_manager.cc:1918] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:26:12.450963 6022 log.cc:1196] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/f56c7274fe9e4d94b0413579737b5ef7
I20260810 07:26:12.451776 6022 ts_tablet_manager.cc:1939] T f56c7274fe9e4d94b0413579737b5ef7 P f1f85a7cd8b64e4488678496aea176ed: Deleting consensus metadata
I20260810 07:26:12.467422 6023 tablet_replica.cc:330] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: stopping tablet replica
I20260810 07:26:12.467725 6023 raft_consensus.cc:2229] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:26:12.467967 6023 raft_consensus.cc:2258] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:26:12.468062 5378 catalog_manager.cc:4841] TS f1f85a7cd8b64e4488678496aea176ed (127.24.60.195:35821): tablet f56c7274fe9e4d94b0413579737b5ef7 (table test-table [id=6988985795b1414ca2ff2fa1c5979e24]) successfully deleted
I20260810 07:26:12.472092 6023 ts_tablet_manager.cc:1905] T f56c7274fe9e4d94b0413579737b5ef7 P de067db565a543c68ecc1ac0dfe8f464: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 07:26:12.473230 5378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:44669 (error 108)
I20260810 07:26:12.473893 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 5666
I20260810 07:26:12.483642 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.483478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:44669 (local address 127.0.0.1:58208)
0810 07:26:12.483532 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.483587 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:44669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 07:26:12.483728 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 5826
W20260810 07:26:12.486562 5378 catalog_manager.cc:4584] TS b900e6d0a5ab4cc7af77c563aea09920 (127.24.60.193:44669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f56c7274fe9e4d94b0413579737b5ef7: Network error: Client connection negotiation failed: client connection to 127.24.60.193:44669: connect: Connection refused (error 111)
I20260810 07:26:12.488129 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:12.487998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:46015 (local address 127.0.0.1:45796)
0810 07:26:12.488056 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:12.488084 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.194:46015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 07:26:12.493407 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 5358
I20260810 07:26:12.505357 24819 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 07:26:12.850934 24819 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 07:26:12.851068 24819 mini_ranger.cc:66] Stopping Ranger...
I20260810 07:26:13.196925 24819 mini_ranger.cc:68] Stopped Ranger
2026-08-10 07:26:13.197 UTC [4860] LOG: received smart shutdown request
2026-08-10 07:26:13.202 UTC [4860] LOG: background worker "logical replication launcher" (PID 4870) exited with exit code 1
2026-08-10 07:26:13.202 UTC [4864] LOG: shutting down
2026-08-10 07:26:13.408 UTC [4860] LOG: database system is shut down
2026-08-10T07:26:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50923 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:26:13 dist-test-slave-vxj2 krb5kdc[6036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:26:13 dist-test-slave-vxj2 krb5kdc[6036](info): set up 2 sockets
Aug 10 07:26:13 dist-test-slave-vxj2 krb5kdc[6036](info): commencing operation
krb5kdc: starting...
W20260810 07:26:14.651971 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:26:14 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:26:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/rangeradmin_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/rangeradmin_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/rangerlookup_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/rangerlookup_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
I20260810 07:26:14.836366 24819 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskghm2j2/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/postgres -l logfile start
2026-08-10 07:26:17.587 UTC [6074] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 07:26:17.587 UTC [6074] LOG: listening on IPv4 address "127.24.60.212", port 57627
2026-08-10 07:26:17.591 UTC [6074] LOG: listening on Unix socket "/tmp/.s.PGSQL.57627"
2026-08-10 07:26:17.631 UTC [6078] LOG: database system was shut down at 2026-08-10 07:26:15 UTC
2026-08-10 07:26:17.635 UTC [6074] LOG: database system is ready to accept connections
I20260810 07:26:18.693382 24819 mini_postgres.cc:96] Postgres bound to 57627
2026-08-10 07:26:18.700 UTC [6087] FATAL: database "slave" does not exist
127.24.60.212:57627 - accepting connections
I20260810 07:26:18.700814 24819 mini_ranger.cc:161] Starting Ranger...
I20260810 07:26:18.717540 24819 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 07:26:19.272827 24819 mini_ranger.cc:87] Created ranger Postgres database
I20260810 07:26:19.272909 24819 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-admin
2026-08-10 07:26:19,579 [I] DB FLAVOR :POSTGRES
2026-08-10 07:26:19,580 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:26:19,580 [I] Checking connection..
2026-08-10 07:26:19,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 07:26:19,898 [I] Checking connection passed.
2026-08-10 07:26:19,898 [I] --------- Verifying version history table ---------
2026-08-10 07:26:19,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:20,176 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:26:20,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:20,452 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:26:20,453 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 07:26:20,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 07:26:20.719 UTC [6160] WARNING: there is no transaction in progress
2026-08-10 07:26:20,729 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 07:26:20,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:21,022 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 07:26:21,022 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:26:21,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 07:26:21,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 07:26:21,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vxj2.c.gcp-upstream.internal','N') ;"
2026-08-10 07:26:21,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:26:22,140 [I] Table x_portal_user does not exist in database ranger
2026-08-10 07:26:22,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:26:22,420 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 07:26:22,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:26:22,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:26:22,955 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 07:26:22,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 07:26:24.028 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.047 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.066 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.084 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.104 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.124 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.188 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.196 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.212 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.225 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.237 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.253 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.264 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.272 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.280 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:24.287 UTC [6289] WARNING: there is no transaction in progress
2026-08-10 07:26:25,051 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 07:26:25,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 07:26:25,333 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 07:26:25,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:26:25,626 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 07:26:25,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:26:25,902 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 07:26:25,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:26:26,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:26:26,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vxj2.c.gcp-upstream.internal';"
2026-08-10 07:26:26,726 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 07:26:26,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 07:26:27,007 [I] DB_PATCHES have already been applied
I20260810 07:26:27.013355 24819 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-admin:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskghm2j2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 07:26:27.013429 24819 mini_ranger.cc:193] Using host: 127.24.60.212
I20260810 07:26:27.016999 24819 mini_ranger.cc:227] Ranger admin URL: http://127.24.60.212:47483
Aug 10, 2026 7:26:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:26:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:26:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:26:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:26:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:26:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:26:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 07:26:28.279067 24819 mini_ranger.cc:160] Time spent starting Ranger: real 9.578s user 0.000s sys 0.010s
Aug 10, 2026 7:26:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:26:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:26:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:26:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:26:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 07:26:52.838526 24819 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.60.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.60.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/rangerkms_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.60.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/rangerkms_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.60.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.60.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.60.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/HTTP_127.24.60.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 07:26:52 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346812, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 07:26:52.939695 24819 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 07:26:52.950371 24819 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 07:26:54.076128 24819 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 07:26:54.076202 24819 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms
I20260810 07:26:54.076216 24819 mini_ranger_kms.cc:227] Using postgres at 127.24.60.212:57627
2026-08-10 07:26:54,254 [I] DB FLAVOR :POSTGRES
2026-08-10 07:26:54,254 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:26:54,254 [I] Checking connection
2026-08-10 07:26:54,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 07:26:54,523 [I] connection success
2026-08-10 07:26:54,523 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 07:26:54,523 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 07:26:54,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 07:26:54,774 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 07:26:54,774 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:26:54,774 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 07:26:54,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.60.212:57627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 07:26:55,051 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 07:26:55.625479 24819 mini_ranger_kms.cc:310] Created kms service
I20260810 07:26:55.843101 24819 mini_ranger_kms.cc:326] Created kudu user
I20260810 07:26:55.898797 24819 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 07:26:55 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346812, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:26:56.327050 24819 mini_ranger_kms.cc:384] Added ranger policy
I20260810 07:26:56.327185 24819 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms:/tmp/dist-test-taskghm2j2/build/release/bin/postgresql.jar:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskghm2j2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 07:26:56.327205 24819 mini_ranger_kms.cc:240] Using host: 127.24.60.212
I20260810 07:26:56.331835 24819 mini_ranger_kms.cc:276] Ranger KMS PID: 6512
I20260810 07:26:56.332026 24819 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.60.212:43753
Aug 10, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:26:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:26:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 07:26:58 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346818, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.60.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:58.873450 24819 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.934s user 0.013s sys 0.146s
I20260810 07:26:58.873569 24819 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 07:26:58.873620 24819 mini_ranger_kms.cc:401] 127.24.60.212:43753/kms/v1/keys
Aug 10 07:26:59 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346818, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.60.212@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
Aug 10 07:26:59 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346818, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.60.212@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
Aug 10 07:27:02 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:27:02.349913 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44631
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:44631
--ranger_config_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:02.460402 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:02.460683 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:02.460716 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:02.462045 6571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:02.462096 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:02.462111 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:02.462124 6571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:02.462138 6571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:02.463924 6571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.60.212:43753/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:44631
--ranger_config_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:02.464196 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:02.464407 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:27:02.470677 6571 server_base.cc:1034] running on GCE node
W20260810 07:27:02.470754 6579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:02.470758 6582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:02.470764 6578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:02.471290 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:02.471693 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:02.472863 6571 hybrid_clock.cc:648] HybridClock initialized: now 1786346822472810 us; error 67 us; skew 500 ppm
Aug 10 07:27:02 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:02.475970 6571 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:02.476553 6571 webserver.cc:460] Webserver started at http://127.24.60.254:43485/ using document root <none> and password file <none>
I20260810 07:27:02.476984 6571 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:02.477083 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:02.477314 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:02 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:02.539615 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "e0a986e78b394b35a9d18d853f92615e"
format_stamp: "Formatted at 2026-08-10 07:27:02 on dist-test-slave-vxj2"
server_key: "1c65442f80125c98a3098cfa126efd2f"
server_key_iv: "6ac9dfdf2f93f6125b2b50b5715f1701"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:02.540083 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "e0a986e78b394b35a9d18d853f92615e"
format_stamp: "Formatted at 2026-08-10 07:27:02 on dist-test-slave-vxj2"
server_key: "1c65442f80125c98a3098cfa126efd2f"
server_key_iv: "6ac9dfdf2f93f6125b2b50b5715f1701"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:02.542047 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 07:27:02.557610 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:02.557847 6571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:02.557955 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "e0a986e78b394b35a9d18d853f92615e"
format_stamp: "Formatted at 2026-08-10 07:27:02 on dist-test-slave-vxj2"
server_key: "1c65442f80125c98a3098cfa126efd2f"
server_key_iv: "6ac9dfdf2f93f6125b2b50b5715f1701"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:02.558081 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:02.594414 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:02.597325 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:02.597543 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:02.603885 6571 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:44631
I20260810 07:27:02.603955 6650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:44631 every 8 connection(s)
I20260810 07:27:02.604526 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:02.605535 6651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:02.608548 6651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Bootstrap starting.
I20260810 07:27:02.609510 6651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:02.609555 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 6571
I20260810 07:27:02.609656 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:27:02.609942 24819 external_mini_cluster.cc:1428] Setting key 364f6e05aa3876b28923a6d03844d705
I20260810 07:27:02.609953 6651 log.cc:824] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:02.610976 6651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: No bootstrap required, opened a new log
I20260810 07:27:02.614137 6651 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:02.614370 6651 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:02.614405 6651 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a986e78b394b35a9d18d853f92615e, State: Initialized, Role: FOLLOWER
I20260810 07:27:02.614545 6651 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:02.614677 6651 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:02.614763 6651 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:02.614825 6651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:02.615525 6651 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:02.615669 6651 leader_election.cc:302] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0a986e78b394b35a9d18d853f92615e; no voters:
I20260810 07:27:02.615875 6651 leader_election.cc:288] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:02.615986 6658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:02.616225 6651 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:02.616240 6658 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 LEADER]: Becoming Leader. State: Replica: e0a986e78b394b35a9d18d853f92615e, State: Running, Role: LEADER
I20260810 07:27:02.616457 6658 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:02.616796 6658 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0a986e78b394b35a9d18d853f92615e. Latest consensus state: current_term: 1 leader_uuid: "e0a986e78b394b35a9d18d853f92615e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } } }
I20260810 07:27:02.616878 6658 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:02.618085 6651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 10 07:27:02 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:02.620757 6660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0a986e78b394b35a9d18d853f92615e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } } }
I20260810 07:27:02.621136 6660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:02.621661 6666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:02.622651 6666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:02.627103 6679 mvcc.cc:205] Tried to move back new op lower bound from 7316876585477431296 to 7316876585437192192. Current Snapshot: MvccSnapshot[applied={T|T < 7316876585477431296}]
I20260810 07:27:02.627878 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:02.610640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.24.60.254:44631)
0810 07:27:02.610866 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:02.610876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:02.610918 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:27:02.611798 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:02.611810 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:02.611913 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:02.612111 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:02.613074 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:02.614629 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:02.615836 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:02.616556 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:02.616680 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:02.620779 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:02.620829 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:02.620830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:02.620947 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:02.624764 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:02.625380 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:02.625384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:02.625385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:02.625505 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:02.625742 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:02.625742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:02.625743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:02.626213 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:02.626423 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:02.627097 (+ 674us) server_negotiation.cc:299] Negotiation successful
0810 07:27:02.627529 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260810 07:27:02.628021 6666 catalog_manager.cc:1347] Generated new cluster ID: ddce46ff2688459ab255317760ed58d7
I20260810 07:27:02.628077 6666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:02.647967 6666 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:02.648895 6666 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:02.659936 6666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Generated new TSK 0
I20260810 07:27:02.660308 6666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:27:03.480990 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44631
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:03.593878 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:03.594122 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:03.594151 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:03.595515 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:03.595573 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:03.595602 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:03.597483 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.60.212:43753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:03.597831 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:03.598145 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:03.604233 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:03.604421 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:03.605077 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:03.605517 6706 server_base.cc:1034] running on GCE node
I20260810 07:27:03.605785 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:03.606096 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:03.607311 6706 hybrid_clock.cc:648] HybridClock initialized: now 1786346823607285 us; error 53 us; skew 500 ppm
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:03.610684 6706 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:03.611234 6706 webserver.cc:460] Webserver started at http://127.24.60.193:46213/ using document root <none> and password file <none>
I20260810 07:27:03.611507 6706 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:03.611584 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:03.611754 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:03.628296 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "82647b41497c48dfa1e988b3b43b9fe2"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "1d47dc4c27195891ef5b7929fa17f8c3"
server_key_iv: "58a43677a3fed1026cc1ddab373ba8ff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:03.629021 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "82647b41497c48dfa1e988b3b43b9fe2"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "1d47dc4c27195891ef5b7929fa17f8c3"
server_key_iv: "58a43677a3fed1026cc1ddab373ba8ff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:03.631238 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 07:27:03.645267 6722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:03.645522 6706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260810 07:27:03.645640 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "82647b41497c48dfa1e988b3b43b9fe2"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "1d47dc4c27195891ef5b7929fa17f8c3"
server_key_iv: "58a43677a3fed1026cc1ddab373ba8ff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:03.645766 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:03.656102 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:03.658797 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:03.659018 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:03.659318 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:03.659759 6706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:03.659832 6706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:03.659899 6706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:03.660029 6706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:03.668988 6706 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36581
I20260810 07:27:03.669060 6857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36581 every 8 connection(s)
I20260810 07:27:03.669576 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:03.670228 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 6706
I20260810 07:27:03.670347 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:27:03.670584 24819 external_mini_cluster.cc:1428] Setting key 376df6660d3372bbc5715303d03dd2e9
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
I20260810 07:27:03.686335 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:03.674553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:41057 (local address 127.24.60.254:44631)
0810 07:27:03.677334 (+ 2781us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:03.677336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:03.677350 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:27:03.677382 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:03.677383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:03.677404 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:03.677520 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:03.678307 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.678864 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.680099 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.680334 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.680428 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:03.682905 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:03.682907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:03.682907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:03.682923 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:03.684619 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:03.685167 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:03.685189 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:03.685190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:03.685263 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:03.685580 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:03.685581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:03.685582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:03.685720 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:03.685798 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:03.686128 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 07:27:03.686227 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2747,"thread_start_us":73,"threads_started":1}
I20260810 07:27:03.686754 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:03.671009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44631 (local address 127.24.60.193:41057)
0810 07:27:03.671192 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:03.671239 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:03.671895 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:03.677636 (+ 5741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:03.677641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:03.677868 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:03.678194 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:03.678202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.678993 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.678994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:03.679979 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:03.679983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.680457 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.680458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:03.680603 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:03.681271 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:03.681280 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:03.682763 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:03.684746 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:03.684757 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:03.684760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:03.685040 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:03.685358 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:03.685359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:03.685359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:03.685498 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:03.685802 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:03.685810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:03.685980 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 07:27:03.686104 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 07:27:03.686281 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260810 07:27:03.687366 6858 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44631
I20260810 07:27:03.687495 6858 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:03.687711 6858 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:03.688328 6609 ts_manager.cc:194] Registered new tserver with Master: 82647b41497c48dfa1e988b3b43b9fe2 (127.24.60.193:36581)
I20260810 07:27:03.688979 6609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:41057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:27:03.722806 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44631
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:03.840525 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:03.840860 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:03.840909 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:03.842916 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:03.843003 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:03.843050 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:03.845690 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.60.212:43753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:03.846019 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:03.846405 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:03.853571 6879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:03.853811 6876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:03.853564 6877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:03.854918 6867 server_base.cc:1034] running on GCE node
I20260810 07:27:03.855162 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:03.855458 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:03.856647 6867 hybrid_clock.cc:648] HybridClock initialized: now 1786346823856596 us; error 65 us; skew 500 ppm
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:03.861982 6867 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:03.862617 6867 webserver.cc:460] Webserver started at http://127.24.60.194:42069/ using document root <none> and password file <none>
I20260810 07:27:03.862975 6867 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:03.863085 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:03.863255 6867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:03.886755 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "9d1cdc486a17864cd9adb89a8c72dfeb"
server_key_iv: "c1db348da3f25f545fe6f7d56c4ee858"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:03.887408 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "9d1cdc486a17864cd9adb89a8c72dfeb"
server_key_iv: "c1db348da3f25f545fe6f7d56c4ee858"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:03.889858 6867 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:03.923663 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:03.923983 6867 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.000s sys 0.000s
I20260810 07:27:03.924093 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "9d1cdc486a17864cd9adb89a8c72dfeb"
server_key_iv: "c1db348da3f25f545fe6f7d56c4ee858"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:03.924199 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:03.947293 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:03.951587 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:03.951840 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:03.952152 6867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:03.952649 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:03.952764 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:03.952837 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:03.952893 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:03.961056 6867 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:32867
I20260810 07:27:03.961663 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:03.963363 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 6867
I20260810 07:27:03.963464 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:27:03.963656 24819 external_mini_cluster.cc:1428] Setting key b736f662403dac66f38792b0a658f5c1
I20260810 07:27:03.965919 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:32867 every 8 connection(s)
Aug 10 07:27:03 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346823, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:03.978143 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:03.967064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:38397 (local address 127.24.60.254:44631)
0810 07:27:03.967114 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:03.967116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:03.967129 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:03.967846 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:03.967848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:03.967866 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:03.967901 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:03.968848 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.969601 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.971100 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.971412 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.971460 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:03.974433 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:03.974435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:03.974435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:03.974445 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:03.976274 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:03.976865 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:03.976866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:03.976867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:03.976942 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:03.977301 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:03.977302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:03.977302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:03.977455 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:03.977500 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:03.977928 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 07:27:03.978045 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:27:03.979038 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:03.966702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44631 (local address 127.24.60.194:38397)
0810 07:27:03.966949 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:03.967010 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:03.967689 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:03.968106 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:03.968111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:03.968396 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:03.968713 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:03.968721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.969728 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.969729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:03.970992 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:03.970995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:03.971583 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:03.971584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:03.971740 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:03.972397 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:03.972409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:03.974299 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:03.976395 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:03.976406 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:03.976409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:03.976752 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:03.977029 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:03.977030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:03.977030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:03.977217 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:03.977550 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:03.977558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:03.977753 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 07:27:03.978565 (+ 812us) client_negotiation.cc:240] Negotiation successful
0810 07:27:03.978759 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
I20260810 07:27:03.979803 7032 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44631
I20260810 07:27:03.979925 7032 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:03.980151 7032 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:03.980630 6609 ts_manager.cc:194] Registered new tserver with Master: dce8af9d7ffb4d7bbe17c22ad1936e4d (127.24.60.194:32867)
I20260810 07:27:03.981141 6609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:38397
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:27:04.017369 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44631
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:04.155649 7038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:04.156073 7038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:04.156194 7038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:04.158371 7038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:04.158500 7038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:04.158600 7038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:04.161284 7038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.60.212:43753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:04.161597 7038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:04.161918 7038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:04.169736 7048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.170675 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.175441 7046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:04.177409 7038 server_base.cc:1034] running on GCE node
I20260810 07:27:04.177836 7038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:04.178158 7038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:04.181428 7038 hybrid_clock.cc:648] HybridClock initialized: now 1786346824181380 us; error 46 us; skew 500 ppm
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:04.185618 7038 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:04.186331 7038 webserver.cc:460] Webserver started at http://127.24.60.195:37167/ using document root <none> and password file <none>
I20260810 07:27:04.186713 7038 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:04.186865 7038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:04.187067 7038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:04.211376 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "8f63126f125b43c08a3bc38218236266"
format_stamp: "Formatted at 2026-08-10 07:27:04 on dist-test-slave-vxj2"
server_key: "ecae36cee563c4bca9490dee09dc9996"
server_key_iv: "08843e3b5fc76d30aa47957ef26fe06d"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:04.212109 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "8f63126f125b43c08a3bc38218236266"
format_stamp: "Formatted at 2026-08-10 07:27:04 on dist-test-slave-vxj2"
server_key: "ecae36cee563c4bca9490dee09dc9996"
server_key_iv: "08843e3b5fc76d30aa47957ef26fe06d"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:04.218668 7038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260810 07:27:04.233702 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.234030 7038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:27:04.234140 7038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "8f63126f125b43c08a3bc38218236266"
format_stamp: "Formatted at 2026-08-10 07:27:04 on dist-test-slave-vxj2"
server_key: "ecae36cee563c4bca9490dee09dc9996"
server_key_iv: "08843e3b5fc76d30aa47957ef26fe06d"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:04.234251 7038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:04.270522 7038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:04.273299 7038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:04.273541 7038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:04.273873 7038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:04.274294 7038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:04.274390 7038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.274473 7038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:04.274530 7038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.283026 7038 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:37461
I20260810 07:27:04.283089 7199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:37461 every 8 connection(s)
I20260810 07:27:04.283586 7038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:04.290071 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 7038
I20260810 07:27:04.290184 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:27:04.290393 24819 external_mini_cluster.cc:1428] Setting key c6841ce4cf49ee96836327c423f6b3bc
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:04.296587 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:04.284435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:57163 (local address 127.24.60.254:44631)
0810 07:27:04.284486 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:04.284488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:04.284737 (+ 249us) server_negotiation.cc:407] Connection header received
0810 07:27:04.285540 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:04.285542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:04.285554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:04.285586 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:04.286450 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.287245 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.288721 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.289019 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.289062 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:04.293028 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:04.293030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:04.293030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:04.293039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:04.294895 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:04.295427 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:04.295428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:04.295429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:04.295488 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:04.295763 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:04.295764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:04.295764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:04.295928 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:04.295970 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:04.296380 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 07:27:04.296482 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:27:04.297465 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:04.284400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44631 (local address 127.24.60.195:57163)
0810 07:27:04.284595 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:04.284645 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:04.285413 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:04.285771 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:04.285775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:04.286037 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:04.286339 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:04.286345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.287373 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.287374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:04.288605 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:04.288608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.289191 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.289192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:04.289351 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:04.290868 (+ 1517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:04.290879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:04.292898 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:04.294996 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:04.295006 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:04.295009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:04.295322 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:04.295556 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:04.295557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:04.295557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:04.295694 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:04.296020 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:04.296026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:04.296201 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 07:27:04.296975 (+ 774us) client_negotiation.cc:240] Negotiation successful
0810 07:27:04.297193 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260810 07:27:04.298265 7202 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44631
I20260810 07:27:04.298388 7202 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.298600 7202 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:04.299058 6609 ts_manager.cc:194] Registered new tserver with Master: 8f63126f125b43c08a3bc38218236266 (127.24.60.195:37461)
I20260810 07:27:04.299563 6609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:57163
I20260810 07:27:04.302042 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:04.302810 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 6706
I20260810 07:27:04.332963 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 6867
I20260810 07:27:04.356748 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 7038
I20260810 07:27:04.378434 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 6571
I20260810 07:27:04.388937 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44631
--webserver_interface=127.24.60.254
--webserver_port=43485
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:44631
--ranger_config_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:04.478029 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:04.478295 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:04.478320 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:04.479481 7206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:04.479538 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:04.479558 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:04.479571 7206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:04.479583 7206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:04.481269 7206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:44631
--ranger_config_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=43485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:04.481489 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:04.481669 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:04.487140 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.487157 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.487157 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:04.487883 7206 server_base.cc:1034] running on GCE node
I20260810 07:27:04.488209 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:04.488569 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:04.489735 7206 hybrid_clock.cc:648] HybridClock initialized: now 1786346824489697 us; error 56 us; skew 500 ppm
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:04.492515 7206 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:04.492952 7206 webserver.cc:460] Webserver started at http://127.24.60.254:43485/ using document root <none> and password file <none>
I20260810 07:27:04.493278 7206 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:04.493366 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:04.537514 7206 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:04.538401 7225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.538648 7206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:04.538757 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "e0a986e78b394b35a9d18d853f92615e"
format_stamp: "Formatted at 2026-08-10 07:27:02 on dist-test-slave-vxj2"
server_key: "1c65442f80125c98a3098cfa126efd2f"
server_key_iv: "6ac9dfdf2f93f6125b2b50b5715f1701"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:04.539193 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:04.556073 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:04.558471 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:04.558643 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:04.563445 7206 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:44631
I20260810 07:27:04.563494 7289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:44631 every 8 connection(s)
I20260810 07:27:04.563934 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:04.565479 7291 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:04.566141 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Bootstrap starting.
I20260810 07:27:04.566622 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 7206
I20260810 07:27:04.566957 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:36581
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:44631
--webserver_port=46213
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:04.575553 7291 log.cc:824] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:04.577601 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:04.577932 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Bootstrap complete.
I20260810 07:27:04.580507 7291 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:04.580844 7291 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a986e78b394b35a9d18d853f92615e, State: Initialized, Role: FOLLOWER
I20260810 07:27:04.581076 7291 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:04.581194 7291 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:04.581254 7291 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:04.581312 7291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:27:04.582465 7291 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:04.582605 7291 leader_election.cc:302] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0a986e78b394b35a9d18d853f92615e; no voters:
I20260810 07:27:04.582834 7291 leader_election.cc:288] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:27:04.582923 7297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:27:04.583109 7297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [term 2 LEADER]: Becoming Leader. State: Replica: e0a986e78b394b35a9d18d853f92615e, State: Running, Role: LEADER
I20260810 07:27:04.583165 7291 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:04.584012 7297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } }
I20260810 07:27:04.584638 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0a986e78b394b35a9d18d853f92615e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } } }
I20260810 07:27:04.584638 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0a986e78b394b35a9d18d853f92615e. Latest consensus state: current_term: 2 leader_uuid: "e0a986e78b394b35a9d18d853f92615e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a986e78b394b35a9d18d853f92615e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44631 } } }
I20260810 07:27:04.584753 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:04.584838 7291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 07:27:04.584751 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:04.585333 7313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:04.588578 7313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:04.589058 7313 catalog_manager.cc:1259] Loaded cluster ID: ddce46ff2688459ab255317760ed58d7
I20260810 07:27:04.589115 7313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:04.590250 7313 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:04.590492 7313 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Loaded TSK: 0
I20260810 07:27:04.590816 7313 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:27:04.690804 7294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:04.691130 7294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:04.691169 7294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:04.693243 7294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:04.693365 7294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:04.693446 7294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:04.695928 7294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:36581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=46213
--tserver_master_addrs=127.24.60.254:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:04.696349 7294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:04.696671 7294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:04.703706 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.704082 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:04.704268 7294 server_base.cc:1034] running on GCE node
W20260810 07:27:04.704834 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:04.705197 7294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:04.705518 7294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:04.706697 7294 hybrid_clock.cc:648] HybridClock initialized: now 1786346824706645 us; error 67 us; skew 500 ppm
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:04.710381 7294 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:04.711040 7294 webserver.cc:460] Webserver started at http://127.24.60.193:46213/ using document root <none> and password file <none>
I20260810 07:27:04.711346 7294 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:04.711431 7294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:04.739161 7294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 07:27:04.740291 7353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.740477 7294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:04.740582 7294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "82647b41497c48dfa1e988b3b43b9fe2"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "1d47dc4c27195891ef5b7929fa17f8c3"
server_key_iv: "58a43677a3fed1026cc1ddab373ba8ff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:04.741137 7294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:04.753355 7294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:04.755705 7294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:04.755894 7294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:04.756139 7294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:04.756516 7294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:04.756572 7294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.756605 7294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:04.756624 7294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.764442 7294 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36581
I20260810 07:27:04.764640 7489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36581 every 8 connection(s)
I20260810 07:27:04.764974 7294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:04.775396 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 7294
I20260810 07:27:04.775677 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:32867
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:44631
--webserver_port=42069
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:04.780427 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:04.765956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44631 (local address 127.24.60.193:58119)
0810 07:27:04.766197 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:04.766251 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:04.766958 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:04.767597 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:04.767602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:04.767845 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:04.768130 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:04.768136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.769917 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.769918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:04.771317 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:04.771321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.772215 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.772217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:04.772378 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:04.773021 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:04.773030 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:04.775156 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:04.778161 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:04.778170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:04.778173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:04.778525 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:04.778886 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:04.778887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:04.778888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:04.779049 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:04.779592 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:04.779601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:04.779786 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 07:27:04.779936 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 07:27:04.780125 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":125,"threads_started":1}
I20260810 07:27:04.781422 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:04.766115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58119 (local address 127.24.60.254:44631)
0810 07:27:04.766327 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:04.766334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:04.766371 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:27:04.767253 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:04.767261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:04.767331 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:04.767525 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:04.768280 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.769786 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.771428 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.772085 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.772203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:04.775333 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:04.775339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:04.775340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:04.775355 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:04.778043 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:04.778640 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:04.778644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:04.778644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:04.778779 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:04.779141 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:04.779142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:04.779142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:04.779494 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:04.779587 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:04.780979 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0810 07:27:04.781122 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260810 07:27:04.782409 7490 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44631
I20260810 07:27:04.782536 7490 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.782768 7490 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
W20260810 07:27:04.783910 7490 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:44631 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 07:27:04.783988 7490 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.784067 7490 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:04.784572 7490 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.784677 7490 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
W20260810 07:27:04.785110 7490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:04.898144 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:04.898456 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:04.898502 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:04.900328 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:04.900414 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:04.900465 7494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:04.902956 7494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:32867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=42069
--tserver_master_addrs=127.24.60.254:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:04.903280 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:04.903592 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:04.910034 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.910158 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:04.910010 7503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:04.911187 7494 server_base.cc:1034] running on GCE node
I20260810 07:27:04.911448 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:04.911758 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:04.912935 7494 hybrid_clock.cc:648] HybridClock initialized: now 1786346824912904 us; error 51 us; skew 500 ppm
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:04.915808 7494 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:04.916414 7494 webserver.cc:460] Webserver started at http://127.24.60.194:42069/ using document root <none> and password file <none>
I20260810 07:27:04.916697 7494 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:04.916790 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:04.944489 7494 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260810 07:27:04.946688 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.946957 7494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 07:27:04.947077 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d"
format_stamp: "Formatted at 2026-08-10 07:27:03 on dist-test-slave-vxj2"
server_key: "9d1cdc486a17864cd9adb89a8c72dfeb"
server_key_iv: "c1db348da3f25f545fe6f7d56c4ee858"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:04.947512 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:04.960772 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:04.963512 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:04.963742 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:04.964042 7494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:04.964432 7494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:04.964497 7494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.964542 7494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:04.964569 7494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:04.976253 7494 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:32867
I20260810 07:27:04.976826 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:04.976806 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:32867 every 8 connection(s)
I20260810 07:27:04.978266 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 7494
I20260810 07:27:04.978595 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.60.212:43753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:37461
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:44631
--webserver_port=37167
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:27:04 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346824, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:04.992959 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:04.977757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:34387 (local address 127.24.60.254:44631)
0810 07:27:04.977839 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:04.977842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:04.978103 (+ 261us) server_negotiation.cc:407] Connection header received
0810 07:27:04.979339 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:04.979341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:04.979361 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:04.979394 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:04.980365 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.981205 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.985576 (+ 4371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.985908 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.985968 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:04.989099 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:04.989101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:04.989101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:04.989110 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:04.991035 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:04.991639 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:04.991640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:04.991641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:04.991706 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:04.992041 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:04.992042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:04.992042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:04.992200 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:04.992248 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:04.992750 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 07:27:04.992850 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:27:04.993567 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:04.977726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44631 (local address 127.24.60.194:34387)
0810 07:27:04.977963 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:04.978007 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:04.979204 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:04.979640 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:04.979644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:04.979919 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:04.980233 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:04.980240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.981330 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.981331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:04.985400 (+ 4069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:04.985413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:04.986122 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:04.986124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:04.986323 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:04.987025 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:04.987037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:04.988957 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:04.991152 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:04.991160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:04.991163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:04.991532 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:04.991791 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:04.991791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:04.991792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:04.991948 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:04.992302 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:04.992309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:04.992549 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 07:27:04.993077 (+ 528us) client_negotiation.cc:240] Negotiation successful
0810 07:27:04.993299 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":129,"threads_started":1}
I20260810 07:27:04.994109 7654 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44631
I20260810 07:27:04.994220 7654 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.994436 7654 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
W20260810 07:27:04.995198 7654 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:44631 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 07:27:04.995283 7654 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.995358 7654 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:04.995754 7654 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:04.995836 7654 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
W20260810 07:27:04.996181 7654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:05.096029 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:05.096366 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:05.096414 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:05.098421 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:05.098508 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:05.098557 7657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:05.101038 7657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.60.212:43753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:37461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=37167
--tserver_master_addrs=127.24.60.254:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:05.101402 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:05.101738 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:05.110670 7668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:05.110702 7663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:05.110670 7665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:05.112084 7657 server_base.cc:1034] running on GCE node
I20260810 07:27:05.112394 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:05.112751 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:05.113955 7657 hybrid_clock.cc:648] HybridClock initialized: now 1786346825113913 us; error 54 us; skew 500 ppm
Aug 10 07:27:05 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:05.117594 7657 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:05.118315 7657 webserver.cc:460] Webserver started at http://127.24.60.195:37167/ using document root <none> and password file <none>
I20260810 07:27:05.118644 7657 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:05.118732 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:05 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:05.137897 7657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 07:27:05.138758 7676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:05.138922 7657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:05.139009 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "8f63126f125b43c08a3bc38218236266"
format_stamp: "Formatted at 2026-08-10 07:27:04 on dist-test-slave-vxj2"
server_key: "ecae36cee563c4bca9490dee09dc9996"
server_key_iv: "08843e3b5fc76d30aa47957ef26fe06d"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:05.140079 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:05.155181 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:05.157701 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:05.157893 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:05.158192 7657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:05.158638 7657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:05.158701 7657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:05.158743 7657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:05.158763 7657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:05.166836 7657 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:37461
I20260810 07:27:05.166919 7813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:37461 every 8 connection(s)
I20260810 07:27:05.167284 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:05.167591 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 7657
Aug 10 07:27:05 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:05.177227 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:05.168271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.24.60.254:44631)
0810 07:27:05.168316 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:05.168318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:05.168331 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:05.168378 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:05.168379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:05.168392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:05.168462 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:05.169256 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:05.170002 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:05.171954 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:05.172225 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:05.172312 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:05.173965 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:05.173966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:05.173967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:05.173976 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:05.175819 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:05.176282 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:05.176282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:05.176283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:05.176329 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:05.176563 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:05.176564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:05.176565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:05.176714 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:05.176912 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:05.177006 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 07:27:05.177101 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:27:05.180685 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:05.168129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44631 (local address 127.24.60.195:43215)
0810 07:27:05.168509 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:05.168561 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:05.169060 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:05.169535 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:05.169540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:05.169766 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:05.170054 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:05.170061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:05.170776 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:05.170778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:05.171649 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:05.171652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:05.172182 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:05.172184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:05.172353 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:05.173037 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:05.173048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:05.176006 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:05.178940 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:05.178949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:05.178952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:05.179209 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:05.179475 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:05.179475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:05.179476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:05.179590 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:05.179890 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:05.179894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:05.180080 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 07:27:05.180205 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 07:27:05.180420 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":88,"threads_started":1}
I20260810 07:27:05.181597 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:05.168373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:43215 (local address 127.24.60.254:44631)
0810 07:27:05.169299 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:05.169301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:05.169309 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:27:05.169344 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:05.169345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:05.169359 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:05.169420 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:05.170136 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:05.170681 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:05.171771 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:05.172058 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:05.172123 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:05.176114 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:05.176116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:05.176117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:05.176124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:05.178844 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:05.179313 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:05.179314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:05.179315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:05.179391 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:05.179667 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:05.179668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:05.179668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:05.179818 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:05.179900 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:05.181363 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0810 07:27:05.181486 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":57,"threads_started":1}
I20260810 07:27:05.182103 7814 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44631
I20260810 07:27:05.182226 7814 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:05.182466 7814 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
W20260810 07:27:05.183316 7814 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:44631 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 07:27:05.183413 7814 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:05.183492 7814 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:05.184588 7814 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:05.184700 7814 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
W20260810 07:27:05.185420 7814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 07:27:05 dist-test-slave-vxj2 krb5kdc[6036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:05.785326 7490 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:05.785509 7490 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:05.786041 7247 ts_manager.cc:194] Registered new tserver with Master: 82647b41497c48dfa1e988b3b43b9fe2 (127.24.60.193:36581)
I20260810 07:27:05.787093 7247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:58119
I20260810 07:27:05.996417 7654 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:05.996628 7654 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:05.997082 7247 ts_manager.cc:194] Registered new tserver with Master: dce8af9d7ffb4d7bbe17c22ad1936e4d (127.24.60.194:32867)
I20260810 07:27:05.997675 7247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:34387
Aug 10 07:27:06 dist-test-slave-vxj2 krb5kdc[6036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346825, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for HTTP/127.24.60.212@KRBTEST.COM
I20260810 07:27:06.186643 7814 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:06.186822 7814 heartbeater.cc:511] Master 127.24.60.254:44631 requested a full tablet report, sending...
I20260810 07:27:06.187233 7247 ts_manager.cc:194] Registered new tserver with Master: 8f63126f125b43c08a3bc38218236266 (127.24.60.195:37461)
I20260810 07:27:06.187765 7247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:43215
I20260810 07:27:06.195550 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:06.213330 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.196674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.24.60.254:44631)
0810 07:27:06.196841 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.196843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.197007 (+ 164us) server_negotiation.cc:407] Connection header received
0810 07:27:06.197037 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.197038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.197051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.197119 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:06.197866 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.198695 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.199871 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.200193 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.200275 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.209583 (+ 9308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:06.209585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:06.209586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:06.209597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:06.211599 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:06.212078 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:06.212079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:06.212080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:06.212140 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:06.212326 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:06.212327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:06.212327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:06.212475 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:06.212567 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.213095 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.213216 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260810 07:27:06.215678 7247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:06.218226 7247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:06.240097 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.226590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.24.60.194:32867)
0810 07:27:06.228276 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.228284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.235727 (+ 7443us) server_negotiation.cc:407] Connection header received
0810 07:27:06.235808 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.235811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.235831 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.235910 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:06.236157 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.237277 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.238668 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.239496 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.239554 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.239618 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.239942 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.239997 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1624,"thread_start_us":77,"threads_started":1}
I20260810 07:27:06.240379 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.226493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:32867 (local address 127.0.0.1:35506)
0810 07:27:06.235622 (+ 9129us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:06.235633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:06.235727 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:06.236024 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:06.236025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:06.236027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:06.236077 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.236079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.237376 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.237377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.238558 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.238561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.239687 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.239688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.239796 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.239815 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:27:06.240217 (+ 402us) client_negotiation.cc:240] Negotiation successful
0810 07:27:06.240301 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9100,"thread_start_us":105,"threads_started":1}
I20260810 07:27:06.241019 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.226118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:37461 (local address 127.0.0.1:57984)
0810 07:27:06.233651 (+ 7533us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:06.233661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:06.233717 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:06.233978 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:06.233981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:06.233984 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:06.234199 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.234202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.237119 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.237120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.238784 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.238787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.240017 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.240018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.240104 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.240110 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:27:06.240865 (+ 755us) client_negotiation.cc:240] Negotiation successful
0810 07:27:06.240925 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7515,"mutex_wait_us":70,"thread_start_us":644,"threads_started":1}
I20260810 07:27:06.241392 7571 tablet_service.cc:1467] Processing CreateTablet for tablet 6c293bd931104f73a767ec3fc6bb653d (DEFAULT_TABLE table=test-table [id=dc8d19cbfe6548baad8834c3b4a1bc36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:06.241701 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.229436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.24.60.195:37461)
0810 07:27:06.233337 (+ 3901us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.233344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.233755 (+ 411us) server_negotiation.cc:407] Connection header received
0810 07:27:06.233817 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.233821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.233843 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.233958 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:06.234297 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.235452 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.238926 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.239780 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.239872 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.239969 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.241479 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.241541 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3849,"thread_start_us":107,"threads_started":1}
I20260810 07:27:06.241844 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c293bd931104f73a767ec3fc6bb653d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:06.243152 7732 tablet_service.cc:1467] Processing CreateTablet for tablet 6c293bd931104f73a767ec3fc6bb653d (DEFAULT_TABLE table=test-table [id=dc8d19cbfe6548baad8834c3b4a1bc36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:06.243609 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c293bd931104f73a767ec3fc6bb653d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:06.246116 7855 tablet_bootstrap.cc:492] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Bootstrap starting.
I20260810 07:27:06.247148 7855 tablet_bootstrap.cc:654] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:06.247288 7856 tablet_bootstrap.cc:492] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Bootstrap starting.
I20260810 07:27:06.247983 7856 tablet_bootstrap.cc:654] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:06.248324 7856 log.cc:824] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:06.248477 7855 log.cc:824] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:06.248984 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.225832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36581 (local address 127.0.0.1:58568)
0810 07:27:06.233317 (+ 7485us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:06.233397 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:06.233501 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:06.234123 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:06.234124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:06.234126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:06.234199 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.234205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.235578 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.235580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.236988 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.236991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.248668 (+ 11677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.248669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.248754 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.248759 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:27:06.248832 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:27:06.248891 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7420,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260810 07:27:06.249027 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.225925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.24.60.193:36581)
0810 07:27:06.233850 (+ 7925us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.233855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.233868 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:06.233907 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.233909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.233926 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.234019 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:06.234328 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.235452 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.247594 (+ 12142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.248542 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.248646 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.248714 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.248866 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.248927 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7883,"thread_start_us":44,"threads_started":1}
I20260810 07:27:06.249958 7856 tablet_bootstrap.cc:492] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: No bootstrap required, opened a new log
I20260810 07:27:06.250043 7856 ts_tablet_manager.cc:1397] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:27:06.249958 7409 tablet_service.cc:1467] Processing CreateTablet for tablet 6c293bd931104f73a767ec3fc6bb653d (DEFAULT_TABLE table=test-table [id=dc8d19cbfe6548baad8834c3b4a1bc36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:06.250349 7409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c293bd931104f73a767ec3fc6bb653d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:06.252959 7856 raft_consensus.cc:348] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.253234 7856 raft_consensus.cc:374] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:06.253286 7856 raft_consensus.cc:729] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f63126f125b43c08a3bc38218236266, State: Initialized, Role: FOLLOWER
I20260810 07:27:06.253482 7856 consensus_queue.cc:260] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.253845 7856 ts_tablet_manager.cc:1428] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 07:27:06.254016 7814 heartbeater.cc:503] Master 127.24.60.254:44631 was elected leader, sending a full tablet report...
I20260810 07:27:06.255154 7862 tablet_bootstrap.cc:492] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: Bootstrap starting.
I20260810 07:27:06.256215 7862 tablet_bootstrap.cc:654] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:06.256681 7862 log.cc:824] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:06.257761 7862 tablet_bootstrap.cc:492] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: No bootstrap required, opened a new log
I20260810 07:27:06.257849 7862 ts_tablet_manager.cc:1397] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:06.257920 7855 tablet_bootstrap.cc:492] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: No bootstrap required, opened a new log
I20260810 07:27:06.257997 7855 ts_tablet_manager.cc:1397] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260810 07:27:06.260507 7862 raft_consensus.cc:348] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.260699 7862 raft_consensus.cc:374] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:06.260730 7862 raft_consensus.cc:729] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82647b41497c48dfa1e988b3b43b9fe2, State: Initialized, Role: FOLLOWER
I20260810 07:27:06.260905 7862 consensus_queue.cc:260] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.261258 7862 ts_tablet_manager.cc:1428] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:06.261122 7855 raft_consensus.cc:348] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.261355 7855 raft_consensus.cc:374] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:06.261394 7855 raft_consensus.cc:729] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce8af9d7ffb4d7bbe17c22ad1936e4d, State: Initialized, Role: FOLLOWER
I20260810 07:27:06.261404 7490 heartbeater.cc:503] Master 127.24.60.254:44631 was elected leader, sending a full tablet report...
I20260810 07:27:06.261755 7855 consensus_queue.cc:260] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.262611 7654 heartbeater.cc:503] Master 127.24.60.254:44631 was elected leader, sending a full tablet report...
I20260810 07:27:06.262590 7855 ts_tablet_manager.cc:1428] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
W20260810 07:27:06.266708 7491 tablet.cc:2369] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:06.342290 7864 raft_consensus.cc:482] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:06.342514 7864 raft_consensus.cc:504] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.343071 7864 leader_election.cc:288] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f63126f125b43c08a3bc38218236266 (127.24.60.195:37461), dce8af9d7ffb4d7bbe17c22ad1936e4d (127.24.60.194:32867)
I20260810 07:27:06.347708 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.343474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:37841 (local address 127.24.60.195:37461)
0810 07:27:06.343520 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.343523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.343633 (+ 110us) server_negotiation.cc:407] Connection header received
0810 07:27:06.343668 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.343669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.343680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.343737 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:06.344065 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.344849 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.346641 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.347226 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.347282 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.347291 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.347573 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.347623 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:27:06.347708 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.343531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58901 (local address 127.24.60.194:32867)
0810 07:27:06.343591 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.343593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.343741 (+ 148us) server_negotiation.cc:407] Connection header received
0810 07:27:06.343842 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.343843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.343854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.343884 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:06.344101 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.344834 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.346617 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.347224 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.347282 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.347291 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.347573 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.347623 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 07:27:06.348047 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.343233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:32867 (local address 127.24.60.193:58901)
0810 07:27:06.343685 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:06.343695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:06.343767 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:06.343960 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:06.343960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:06.343962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:06.344020 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.344022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.344958 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.344959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.346426 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.346428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.347386 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.347387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.347476 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.347479 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:27:06.347874 (+ 395us) client_negotiation.cc:240] Negotiation successful
0810 07:27:06.347940 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":111,"thread_start_us":129,"threads_started":1}
I20260810 07:27:06.348467 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.343230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:37461 (local address 127.24.60.193:37841)
0810 07:27:06.343516 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:06.343529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:06.343593 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:06.343861 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:06.343862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:06.343864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:06.343966 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.343969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.345017 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.345018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.346423 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:06.346428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.347396 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.347397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:06.347470 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.347474 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:06.348313 (+ 839us) client_negotiation.cc:240] Negotiation successful
0810 07:27:06.348370 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":32,"thread_start_us":130,"threads_started":1}
I20260810 07:27:06.348459 7595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd931104f73a767ec3fc6bb653d" candidate_uuid: "82647b41497c48dfa1e988b3b43b9fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" is_pre_election: true
I20260810 07:27:06.348663 7595 raft_consensus.cc:2454] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82647b41497c48dfa1e988b3b43b9fe2 in term 0.
I20260810 07:27:06.348922 7357 leader_election.cc:302] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82647b41497c48dfa1e988b3b43b9fe2, dce8af9d7ffb4d7bbe17c22ad1936e4d; no voters:
I20260810 07:27:06.349082 7864 raft_consensus.cc:2781] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:06.349133 7864 raft_consensus.cc:482] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:06.349085 7759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd931104f73a767ec3fc6bb653d" candidate_uuid: "82647b41497c48dfa1e988b3b43b9fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f63126f125b43c08a3bc38218236266" is_pre_election: true
I20260810 07:27:06.349180 7864 raft_consensus.cc:3037] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:06.349259 7759 raft_consensus.cc:2454] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82647b41497c48dfa1e988b3b43b9fe2 in term 0.
I20260810 07:27:06.350087 7864 raft_consensus.cc:504] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.350271 7864 leader_election.cc:288] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f63126f125b43c08a3bc38218236266 (127.24.60.195:37461), dce8af9d7ffb4d7bbe17c22ad1936e4d (127.24.60.194:32867)
I20260810 07:27:06.350502 7595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd931104f73a767ec3fc6bb653d" candidate_uuid: "82647b41497c48dfa1e988b3b43b9fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d"
I20260810 07:27:06.350517 7759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd931104f73a767ec3fc6bb653d" candidate_uuid: "82647b41497c48dfa1e988b3b43b9fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f63126f125b43c08a3bc38218236266"
I20260810 07:27:06.350590 7759 raft_consensus.cc:3037] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:06.350600 7595 raft_consensus.cc:3037] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:06.351548 7759 raft_consensus.cc:2454] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82647b41497c48dfa1e988b3b43b9fe2 in term 1.
I20260810 07:27:06.351572 7595 raft_consensus.cc:2454] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82647b41497c48dfa1e988b3b43b9fe2 in term 1.
I20260810 07:27:06.351754 7357 leader_election.cc:302] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82647b41497c48dfa1e988b3b43b9fe2, dce8af9d7ffb4d7bbe17c22ad1936e4d; no voters:
I20260810 07:27:06.351866 7864 raft_consensus.cc:2781] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:06.352049 7864 raft_consensus.cc:686] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 82647b41497c48dfa1e988b3b43b9fe2, State: Running, Role: LEADER
I20260810 07:27:06.352171 7864 consensus_queue.cc:237] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } }
I20260810 07:27:06.353179 7247 catalog_manager.cc:5495] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82647b41497c48dfa1e988b3b43b9fe2 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "82647b41497c48dfa1e988b3b43b9fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82647b41497c48dfa1e988b3b43b9fe2" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20260810 07:27:06.378510 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.375762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.24.60.193:36581)
0810 07:27:06.375813 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:06.375815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:06.375832 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:27:06.375883 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:06.375884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:06.375908 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:06.376039 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:06.376215 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.376737 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.377621 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:06.377825 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:06.377957 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:06.378142 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:06.378264 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:06.378374 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:06.378393 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 07:27:06.378416 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 07:27:06.382154 7759 raft_consensus.cc:1261] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 1 FOLLOWER]: Refusing update from remote peer 82647b41497c48dfa1e988b3b43b9fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:06.382158 7595 raft_consensus.cc:1261] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 1 FOLLOWER]: Refusing update from remote peer 82647b41497c48dfa1e988b3b43b9fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:06.382575 7864 consensus_queue.cc:1035] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce8af9d7ffb4d7bbe17c22ad1936e4d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:06.382716 7868 consensus_queue.cc:1035] T 6c293bd931104f73a767ec3fc6bb653d P 82647b41497c48dfa1e988b3b43b9fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f63126f125b43c08a3bc38218236266" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:06.392148 7875 mvcc.cc:205] Tried to move back new op lower bound from 7316876600858054656 to 7316876600739250176. Current Snapshot: MvccSnapshot[applied={T|T < 7316876600858054656}]
I20260810 07:27:06.392599 7873 mvcc.cc:205] Tried to move back new op lower bound from 7316876600858054656 to 7316876600739250176. Current Snapshot: MvccSnapshot[applied={T|T < 7316876600858054656}]
I20260810 07:27:06.392831 7874 mvcc.cc:205] Tried to move back new op lower bound from 7316876600858054656 to 7316876600739250176. Current Snapshot: MvccSnapshot[applied={T|T < 7316876600858054656 or (T in {7316876600858054656})}]
I20260810 07:27:06.395825 7245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:06.395957 7245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:06.396929 7245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0a986e78b394b35a9d18d853f92615e: Sending DeleteTablet for 3 replicas of tablet 6c293bd931104f73a767ec3fc6bb653d
I20260810 07:27:06.397569 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c293bd931104f73a767ec3fc6bb653d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:06 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:57984
I20260810 07:27:06.397907 7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c293bd931104f73a767ec3fc6bb653d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:06 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:35506
I20260810 07:27:06.398311 7409 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c293bd931104f73a767ec3fc6bb653d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:06 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:58568
I20260810 07:27:06.399175 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 7294
I20260810 07:27:06.404232 7886 tablet_replica.cc:330] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: stopping tablet replica
I20260810 07:27:06.404518 7886 raft_consensus.cc:2229] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:06.404857 7886 raft_consensus.cc:2258] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:06.406316 7886 ts_tablet_manager.cc:1905] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:06.408780 7886 ts_tablet_manager.cc:1918] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:06.408895 7886 log.cc:1196] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/6c293bd931104f73a767ec3fc6bb653d
I20260810 07:27:06.409246 7886 ts_tablet_manager.cc:1939] T 6c293bd931104f73a767ec3fc6bb653d P dce8af9d7ffb4d7bbe17c22ad1936e4d: Deleting consensus metadata
I20260810 07:27:06.409322 7885 tablet_replica.cc:330] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: stopping tablet replica
I20260810 07:27:06.409489 7885 raft_consensus.cc:2229] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:06.409637 7885 raft_consensus.cc:2258] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:06.409801 7230 catalog_manager.cc:4841] TS dce8af9d7ffb4d7bbe17c22ad1936e4d (127.24.60.194:32867): tablet 6c293bd931104f73a767ec3fc6bb653d (table test-table [id=dc8d19cbfe6548baad8834c3b4a1bc36]) successfully deleted
I20260810 07:27:06.410283 7885 ts_tablet_manager.cc:1905] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 07:27:06.411805 7228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:36581 (error 108)
I20260810 07:27:06.412149 7885 ts_tablet_manager.cc:1918] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:06.412225 7885 log.cc:1196] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/6c293bd931104f73a767ec3fc6bb653d
I20260810 07:27:06.412253 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 7494
I20260810 07:27:06.415172 7885 ts_tablet_manager.cc:1939] T 6c293bd931104f73a767ec3fc6bb653d P 8f63126f125b43c08a3bc38218236266: Deleting consensus metadata
I20260810 07:27:06.422681 7226 catalog_manager.cc:4841] TS 8f63126f125b43c08a3bc38218236266 (127.24.60.195:37461): tablet 6c293bd931104f73a767ec3fc6bb653d (table test-table [id=dc8d19cbfe6548baad8834c3b4a1bc36]) successfully deleted
I20260810 07:27:06.423413 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:06.423235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36581 (local address 127.0.0.1:58586)
0810 07:27:06.423300 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:06.423362 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 07:27:06.423695 7228 catalog_manager.cc:4584] TS 82647b41497c48dfa1e988b3b43b9fe2 (127.24.60.193:36581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c293bd931104f73a767ec3fc6bb653d: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36581: connect: Connection refused (error 111)
I20260810 07:27:06.427860 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 7657
I20260810 07:27:06.443871 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 7206
I20260810 07:27:06.455206 24819 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 07:27:06.801025 24819 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 07:27:06.801137 24819 mini_ranger.cc:66] Stopping Ranger...
I20260810 07:27:07.147333 24819 mini_ranger.cc:68] Stopped Ranger
2026-08-10 07:27:07.147 UTC [6074] LOG: received smart shutdown request
2026-08-10 07:27:07.153 UTC [6074] LOG: background worker "logical replication launcher" (PID 6085) exited with exit code 1
2026-08-10 07:27:07.154 UTC [6080] LOG: shutting down
2026-08-10 07:27:07.379 UTC [6074] LOG: database system is shut down
2026-08-10T07:27:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53975 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:07 dist-test-slave-vxj2 krb5kdc[7899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:07 dist-test-slave-vxj2 krb5kdc[7899](info): set up 2 sockets
Aug 10 07:27:07 dist-test-slave-vxj2 krb5kdc[7899](info): commencing operation
krb5kdc: starting...
W20260810 07:27:08.636155 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:27:08 dist-test-slave-vxj2 krb5kdc[7899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:27:08.776378 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33935
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33935,127.24.60.253:40557,127.24.60.252:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:08.863740 7917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:08.863998 7917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:08.864025 7917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:08.865343 7917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:08.865391 7917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:08.865406 7917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:08.865420 7917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:08.865432 7917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:08.867100 7917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33935,127.24.60.253:40557,127.24.60.252:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:08.867400 7917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:08.867658 7917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:08.873310 7923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:08.873243 7926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:08.873227 7924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:08.873385 7917 server_base.cc:1034] running on GCE node
I20260810 07:27:08.873898 7917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:08.874274 7917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:08.875466 7917 hybrid_clock.cc:648] HybridClock initialized: now 1786346828875441 us; error 47 us; skew 500 ppm
Aug 10 07:27:08 dist-test-slave-vxj2 krb5kdc[7899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:08.878364 7917 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:08.878803 7917 webserver.cc:460] Webserver started at http://127.24.60.254:38393/ using document root <none> and password file <none>
I20260810 07:27:08.879045 7917 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:08.879093 7917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:08.879244 7917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:08.880427 7917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "96d92e561e574eeaaf7eb811228c80e4"
format_stamp: "Formatted at 2026-08-10 07:27:08 on dist-test-slave-vxj2"
server_key: "f28c916115526615647daf03c2c87070"
server_key_iv: "3898609bf5506046610600c537e5e4c8"
server_key_version: "encryptionkey@0"
I20260810 07:27:08.880815 7917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "96d92e561e574eeaaf7eb811228c80e4"
format_stamp: "Formatted at 2026-08-10 07:27:08 on dist-test-slave-vxj2"
server_key: "f28c916115526615647daf03c2c87070"
server_key_iv: "3898609bf5506046610600c537e5e4c8"
server_key_version: "encryptionkey@0"
I20260810 07:27:08.882495 7917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:27:08.883546 7933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:08.883936 7917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:08.884064 7917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "96d92e561e574eeaaf7eb811228c80e4"
format_stamp: "Formatted at 2026-08-10 07:27:08 on dist-test-slave-vxj2"
server_key: "f28c916115526615647daf03c2c87070"
server_key_iv: "3898609bf5506046610600c537e5e4c8"
server_key_version: "encryptionkey@0"
I20260810 07:27:08.884155 7917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:08.893230 7917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:08.896037 7917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:08.896188 7917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:08.900975 7917 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:33935
I20260810 07:27:08.901052 8003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:33935 every 8 connection(s)
I20260810 07:27:08.901467 7917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:08.902311 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 7917
I20260810 07:27:08.902468 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:27:08.902588 8004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:08.902720 24819 external_mini_cluster.cc:1428] Setting key d8a6bb4b3f784c3f4e578529e8e25a5a
I20260810 07:27:08.904459 8004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.60.253@KRBTEST.COM; defaulting to no policy
Aug 10 07:27:08 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.253@KRBTEST.COM" created.
I20260810 07:27:08.918339 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:08.905492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33935 (local address 127.0.0.1:44462)
0810 07:27:08.905698 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:08.905743 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:08.906645 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:08.907054 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:08.907058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:08.907548 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:08.907839 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:08.907848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:08.909011 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:08.909013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:08.910436 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:08.910440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:08.910984 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:08.910986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:08.911086 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:08.911653 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:08.911666 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:08.913716 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:08.916096 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:08.916107 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:08.916111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:08.916476 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:08.916893 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:08.916894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:08.916894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:08.916994 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:08.917562 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:08.917572 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:08.917660 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 07:27:08.917768 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 07:27:08.918095 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":149,"thread_start_us":127,"threads_started":1}
I20260810 07:27:08.918350 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:08.905501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.24.60.254:33935)
0810 07:27:08.905695 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:08.905703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:08.905891 (+ 188us) server_negotiation.cc:407] Connection header received
0810 07:27:08.906892 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:08.906894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:08.906923 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:08.907003 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:08.907983 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:08.908850 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:08.910529 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:08.910863 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:08.910960 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:08.913861 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:08.913871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:08.913871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:08.913879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:08.915977 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:08.916595 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:08.916600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:08.916600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:08.916738 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:08.917067 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:08.917068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:08.917069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:08.917463 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:08.917555 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:08.917827 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 07:27:08.918004 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260810 07:27:08.919404 8004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:08.919842 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:08.919678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34738)
0810 07:27:08.919747 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:08.919805 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 07:27:08.920060 7938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
I20260810 07:27:08.920621 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:08.920494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34748)
0810 07:27:08.920551 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:08.920586 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 07:27:08.920825 8004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:40557: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab.
Entry for principal kudu/127.24.60.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab.
I20260810 07:27:08.937105 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:40557
--webserver_interface=127.24.60.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab
--principal=kudu/127.24.60.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33935,127.24.60.253:40557,127.24.60.252:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:08.970101 8004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } attempt: 1
I20260810 07:27:08.970726 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:08.970569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34758)
0810 07:27:08.970639 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:08.970690 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 07:27:08.971222 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:08.971114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34768)
0810 07:27:08.971164 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:08.971194 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 07:27:08.971379 8004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:40557: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
W20260810 07:27:09.026671 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:09.026902 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:09.026937 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:09.028169 8011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:09.028220 8011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:09.028235 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:09.028249 8011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:09.028309 8011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:09.030035 8011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33935,127.24.60.253:40557,127.24.60.252:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab
--principal=kudu/127.24.60.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:40557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.60.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:09.030355 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:09.030625 8011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:09.035944 8017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:09.035971 8018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:09.035997 8020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:09.036266 8011 server_base.cc:1034] running on GCE node
I20260810 07:27:09.036626 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:09.036957 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:09.038127 8011 hybrid_clock.cc:648] HybridClock initialized: now 1786346829038106 us; error 42 us; skew 500 ppm
I20260810 07:27:09.039609 8004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } attempt: 2
I20260810 07:27:09.040042 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.039888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34784)
0810 07:27:09.039962 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.040006 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 07:27:09.040482 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.040347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34788)
0810 07:27:09.040404 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.040446 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 07:27:09.040602 8004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:40557: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40557: connect: Connection refused (error 111)
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:09.041221 8011 init.cc:377] Logged in from keytab as kudu/127.24.60.253@KRBTEST.COM (short username kudu)
I20260810 07:27:09.041734 8011 webserver.cc:460] Webserver started at http://127.24.60.253:39091/ using document root <none> and password file <none>
I20260810 07:27:09.041965 8011 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:09.042009 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:09.042106 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:09.043228 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data/instance:
uuid: "e3ea0a3c0d23496994df531bf3bca82b"
format_stamp: "Formatted at 2026-08-10 07:27:09 on dist-test-slave-vxj2"
server_key: "b3c3dc002f266bfb309d40f334b28ddf"
server_key_iv: "061095c1db94b47c7f095ffb8d8df016"
server_key_version: "encryptionkey@0"
I20260810 07:27:09.043545 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal/instance:
uuid: "e3ea0a3c0d23496994df531bf3bca82b"
format_stamp: "Formatted at 2026-08-10 07:27:09 on dist-test-slave-vxj2"
server_key: "b3c3dc002f266bfb309d40f334b28ddf"
server_key_iv: "061095c1db94b47c7f095ffb8d8df016"
server_key_version: "encryptionkey@0"
I20260810 07:27:09.045081 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:27:09.046106 8029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:09.046371 8011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:09.046495 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal
uuid: "e3ea0a3c0d23496994df531bf3bca82b"
format_stamp: "Formatted at 2026-08-10 07:27:09 on dist-test-slave-vxj2"
server_key: "b3c3dc002f266bfb309d40f334b28ddf"
server_key_iv: "061095c1db94b47c7f095ffb8d8df016"
server_key_version: "encryptionkey@0"
I20260810 07:27:09.046597 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:09.065027 8011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:09.067647 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:09.067849 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:09.073611 8011 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.253:40557
I20260810 07:27:09.073669 8092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.253:40557 every 8 connection(s)
I20260810 07:27:09.074118 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
I20260810 07:27:09.075186 8093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:09.076933 8093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:09.083603 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8011
I20260810 07:27:09.083753 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-1/wal/instance
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:09.084031 24819 external_mini_cluster.cc:1428] Setting key 99e9f62a050c41d11ab76ad91e98a7f5
I20260810 07:27:09.088465 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.077900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.24.60.254:33935)
0810 07:27:09.077978 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.077980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.078175 (+ 195us) server_negotiation.cc:407] Connection header received
0810 07:27:09.078692 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.078693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.078707 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.078817 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.079516 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.080147 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.081336 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.081558 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.081618 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.084738 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.084740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.084740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.084748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.086753 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.087225 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.087226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.087227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.087296 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.087555 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.087555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.087556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.087720 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.087795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.088177 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.088308 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:27:09.088541 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.077798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33935 (local address 127.0.0.1:44468)
0810 07:27:09.078026 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.078073 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.078583 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.078864 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.078867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.079167 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.079393 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.079398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.080277 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.080279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.081215 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.081219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.081672 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.081673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.081811 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.082469 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.082479 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.084593 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.086857 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.086866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.086868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.087153 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.087370 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.087371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.087371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.087485 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.087831 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.087837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.087993 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.088098 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.088323 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":132,"threads_started":1}
I20260810 07:27:09.089257 8093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.60.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.252@KRBTEST.COM" created.
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:27:09.101125 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.089574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34802)
0810 07:27:09.089902 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.089913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.090005 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.090192 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.090193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.090356 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.090411 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.090413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.091361 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.091362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.091887 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.091889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.092406 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.092407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.092479 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.092961 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.092966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.097523 (+ 4557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.099950 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.099951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.099952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.100142 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.100418 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.100419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.100419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.100477 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.100851 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.100852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.100897 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.100955 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.101046 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 07:27:09.101361 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.089655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.24.60.253:40557)
0810 07:27:09.089845 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.089850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.090001 (+ 151us) server_negotiation.cc:407] Connection header received
0810 07:27:09.090081 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.090084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.090099 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.090193 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.090491 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.091230 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.091995 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.092314 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.092407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.097672 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.097687 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.097687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.097696 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.099794 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.100238 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.100242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.100243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.100320 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.100542 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.100543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.100544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.100778 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.100877 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.101021 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.101125 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":203,"threads_started":1}
I20260810 07:27:09.102329 8093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:09.102814 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.102658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42520)
0810 07:27:09.102718 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.102778 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 07:27:09.103008 8033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
I20260810 07:27:09.103545 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.103423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42536)
0810 07:27:09.103477 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.103512 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 07:27:09.103716 8093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:46613: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab.
Entry for principal kudu/127.24.60.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab.
I20260810 07:27:09.118005 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:46613
--webserver_interface=127.24.60.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab
--principal=kudu/127.24.60.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33935,127.24.60.253:40557,127.24.60.252:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:09.131759 8004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } attempt: 3
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:27:09.139736 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.132190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34816)
0810 07:27:09.132256 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.132269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.132357 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.132518 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.132519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.132709 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.132781 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.132783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.133625 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.133626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.134178 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.134180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.134605 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.134606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.134699 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.135238 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.135244 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.136559 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.138538 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.138540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.138540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.138730 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.139003 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.139004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.139005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.139075 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.139370 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.139372 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.139440 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.139517 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.139631 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:27:09.139736 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.132295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.24.60.253:40557)
0810 07:27:09.132375 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.132377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.132389 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:27:09.132422 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.132422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.132438 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.132510 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.132912 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.133513 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.134271 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.134506 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.134592 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.136656 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.136658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.136658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.136674 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.138402 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.138813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.138815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.138815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.138926 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.139153 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.139154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.139155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.139293 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.139367 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.139543 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.139635 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:27:09.140313 8004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:09.140764 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.140657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42552)
0810 07:27:09.140711 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.140743 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 07:27:09.141117 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.141016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42568)
0810 07:27:09.141067 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.141092 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 07:27:09.141229 8004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:46613: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
I20260810 07:27:09.152987 8093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } attempt: 1
I20260810 07:27:09.153560 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.153405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42580)
0810 07:27:09.153471 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.153523 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 07:27:09.154078 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.153960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42596)
0810 07:27:09.154018 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.154052 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 07:27:09.154215 8093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:46613: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
I20260810 07:27:09.172410 8004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } attempt: 1
I20260810 07:27:09.173013 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.172858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42600)
0810 07:27:09.172934 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.172980 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 07:27:09.173441 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.173328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42612)
0810 07:27:09.173381 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.173407 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 07:27:09.173545 8004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:46613: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
W20260810 07:27:09.207515 8101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:09.207811 8101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:09.207865 8101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:09.209100 8101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:09.209203 8101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:09.209242 8101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:09.209267 8101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:09.209297 8101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:09.211050 8101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33935,127.24.60.253:40557,127.24.60.252:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab
--principal=kudu/127.24.60.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.60.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:09.211372 8101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:09.211630 8101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:09.217461 8106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:09.217470 8109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:09.217470 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:09.217797 8101 server_base.cc:1034] running on GCE node
I20260810 07:27:09.218086 8101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:09.218426 8101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:09.219632 8101 hybrid_clock.cc:648] HybridClock initialized: now 1786346829219605 us; error 38 us; skew 500 ppm
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:09.222399 8093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } attempt: 2
I20260810 07:27:09.222817 8101 init.cc:377] Logged in from keytab as kudu/127.24.60.252@KRBTEST.COM (short username kudu)
I20260810 07:27:09.222851 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.222705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42622)
0810 07:27:09.222768 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.222801 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 07:27:09.223263 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.223122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42632)
0810 07:27:09.223187 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.223228 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 07:27:09.223394 8093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:46613: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
I20260810 07:27:09.223474 8101 webserver.cc:460] Webserver started at http://127.24.60.252:39523/ using document root <none> and password file <none>
I20260810 07:27:09.223716 8101 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:09.223767 8101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:09.223870 8101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:09.224968 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data/instance:
uuid: "80109bc4c6104a41bccdbc250cbe9ba4"
format_stamp: "Formatted at 2026-08-10 07:27:09 on dist-test-slave-vxj2"
server_key: "97fc36db01f41ec5477938017a57605c"
server_key_iv: "94ac531ceb7f29b1699c9f260292809f"
server_key_version: "encryptionkey@0"
I20260810 07:27:09.225410 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal/instance:
uuid: "80109bc4c6104a41bccdbc250cbe9ba4"
format_stamp: "Formatted at 2026-08-10 07:27:09 on dist-test-slave-vxj2"
server_key: "97fc36db01f41ec5477938017a57605c"
server_key_iv: "94ac531ceb7f29b1699c9f260292809f"
server_key_version: "encryptionkey@0"
I20260810 07:27:09.226943 8101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:09.227767 8125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:09.228152 8101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:09.228266 8101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal
uuid: "80109bc4c6104a41bccdbc250cbe9ba4"
format_stamp: "Formatted at 2026-08-10 07:27:09 on dist-test-slave-vxj2"
server_key: "97fc36db01f41ec5477938017a57605c"
server_key_iv: "94ac531ceb7f29b1699c9f260292809f"
server_key_version: "encryptionkey@0"
I20260810 07:27:09.228355 8101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:09.248720 8004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } attempt: 2
I20260810 07:27:09.249291 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.249124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42644)
0810 07:27:09.249215 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.249259 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 07:27:09.249958 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.249809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42658)
0810 07:27:09.249900 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.249935 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 07:27:09.250061 8004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:46613: Network error: Client connection negotiation failed: client connection to 127.24.60.252:46613: connect: Connection refused (error 111)
I20260810 07:27:09.255187 8101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:09.258262 8101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:09.258476 8101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:09.263693 8101 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.252:46613
I20260810 07:27:09.263744 8188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.252:46613 every 8 connection(s)
I20260810 07:27:09.264211 8101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
I20260810 07:27:09.265384 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:09.267203 8189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:09.274284 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8101
I20260810 07:27:09.274425 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346663269703-24819-0/minicluster-data/master-2/wal/instance
I20260810 07:27:09.274644 24819 external_mini_cluster.cc:1428] Setting key bdd61cf12bde34ef6d53122b507d4a76
I20260810 07:27:09.278438 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.268095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.24.60.254:33935)
0810 07:27:09.268176 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.268178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.268436 (+ 258us) server_negotiation.cc:407] Connection header received
0810 07:27:09.269033 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.269035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.269050 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.269117 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.269992 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.270522 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.271652 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.271864 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.271967 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.274657 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.274658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.274659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.274669 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.276613 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.277145 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.277146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.277146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.277293 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.277622 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.277623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.277623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.277798 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.277899 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.278219 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.278320 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:27:09.278595 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.268054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33935 (local address 127.0.0.1:44476)
0810 07:27:09.268297 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.268339 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.268890 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.269233 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.269237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.269547 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.269865 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.269870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.270660 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.270665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.271521 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.271523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.271980 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.271981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.272126 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.272672 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.272683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.274547 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.276695 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.276701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.276704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.277032 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.277369 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.277370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.277371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.277532 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.277918 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.277925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.278065 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.278169 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.278323 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":116,"threads_started":1}
I20260810 07:27:09.279317 8189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:27:09.285408 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.275214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.24.60.252:46613)
0810 07:27:09.275444 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.275450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.275470 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:27:09.275503 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.275505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.275517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.275585 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.276415 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.277470 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.278613 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.279096 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.279172 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.282251 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.282261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.282261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.282268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.284111 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.284512 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.284518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.284519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.284577 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.284787 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.284789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.284789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.284951 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.285121 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.285202 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.285286 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":113,"threads_started":1}
I20260810 07:27:09.288206 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.279662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40557 (local address 127.0.0.1:34828)
0810 07:27:09.279728 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.279738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.279774 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.280077 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.280078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.280211 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.280261 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.280262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.281232 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.281233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.282002 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.282005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.282464 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.282465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.282562 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.282922 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.282927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.284459 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.286769 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.286770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.286771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.287030 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.287322 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.287323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.287323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.287401 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.287917 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.287918 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.287957 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.288007 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.288111 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 07:27:09.288273 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.279839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.24.60.253:40557)
0810 07:27:09.279911 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.279913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.279924 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:09.279952 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.279953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.279964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.280033 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.280353 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.281095 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.282093 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.282385 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.282442 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.284567 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.284569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.284569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.284578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.286683 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.287157 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.287159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.287159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.287236 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.287615 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.287616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.287616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.287841 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.287917 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.288056 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.288151 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:27:09.288642 8189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:27:09.295153 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.286363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.24.60.254:33935)
0810 07:27:09.286451 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.286453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.286464 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:09.286495 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.286495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.286522 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.286607 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.287252 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.288028 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.288924 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.289266 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.289341 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.292162 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.292163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.292163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.292171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.293844 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.294207 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.294208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.294208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.294272 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.294498 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.294499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.294499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.294632 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.294874 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.294914 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.295002 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:27:09.296376 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.289057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.24.60.252:46613)
0810 07:27:09.289145 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.289176 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.289185 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:27:09.289205 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.289206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.289218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.289302 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.289638 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.290246 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.291179 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.291514 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.291598 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.293605 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.293606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.293606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.293612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.295081 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.295477 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.295478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.295478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.295543 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.295774 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.295775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.295775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.295933 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.296012 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.296183 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.296287 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":67}
I20260810 07:27:09.296424 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.288937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42680)
0810 07:27:09.288994 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.289024 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.289079 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.289323 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.289324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.289500 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.289543 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.289545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.290388 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.290389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.291087 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.291089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.291605 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.291606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.291675 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.292105 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.292109 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.293520 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.295167 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.295168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.295168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.295395 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.295613 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.295613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.295614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.295684 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.296013 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.296014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.296061 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.296134 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.296238 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:27:09.298398 8189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4: Bootstrap starting.
I20260810 07:27:09.299597 8189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:09.299995 8189 log.cc:824] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4: Log is configured to *not* fsync() on all Append() calls
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:27:09.300853 8189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4: No bootstrap required, opened a new log
I20260810 07:27:09.303278 8189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.303491 8189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:09.303532 8189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80109bc4c6104a41bccdbc250cbe9ba4, State: Initialized, Role: FOLLOWER
I20260810 07:27:09.303699 8189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.304072 8189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:09.304395 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.295922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.24.60.253:40557)
0810 07:27:09.295966 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.295968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.295976 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:27:09.296027 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.296028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.296038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.296086 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.296723 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.297387 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.298303 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.298548 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.298631 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.301316 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.301318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.301318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.301327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.302978 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.303334 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.303336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.303336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.303409 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.303639 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.303641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.303641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.303775 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.304147 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.304205 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.304287 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:27:09.304438 8197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.304524 8197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:27:09.306046 8208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:27:09.306149 8208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:27:09.306263 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:27:09.306589 8101 master_runner.cc:186] Error getting master registration for 127.24.60.254:33935: OK, instance_id { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" instance_seqno: 1786346828899518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:27:09.309103 8101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:27:09.314565 8093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } attempt: 3
I20260810 07:27:09.317764 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.307704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.24.60.253:40557)
0810 07:27:09.307749 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.307751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.307906 (+ 155us) server_negotiation.cc:407] Connection header received
0810 07:27:09.308129 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.308130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.308142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.308174 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.308913 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.309729 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.310843 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.311154 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.311226 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.313363 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.313364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.313365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.313373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.315256 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.316640 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.316642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.316643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.316705 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.316904 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.316904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.316905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.317035 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.317170 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.317559 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.317665 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 07:27:09.319733 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.307950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.24.60.254:33935)
0810 07:27:09.307996 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.307998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.309021 (+ 1023us) server_negotiation.cc:407] Connection header received
0810 07:27:09.309305 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.309306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.309319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.309349 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.310075 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.310842 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.311797 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.312096 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.312171 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.313688 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.313690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.313690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.313699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.316139 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.316993 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.316995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.316995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.317067 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.318113 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.318115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.318116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.318755 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.318906 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.319543 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.319637 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346829, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:27:09.320544 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.308231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.24.60.252:46613)
0810 07:27:09.308275 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.308277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.309130 (+ 853us) server_negotiation.cc:407] Connection header received
0810 07:27:09.310117 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.310119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.310132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.310177 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.310900 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.311974 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.313489 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.314016 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.314067 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.315895 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.315896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.315897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.315905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.317993 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.319277 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.319278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.319278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.319343 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.319569 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.319570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.319571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.319754 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.319868 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.320335 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.320421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:27:09.324182 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.314857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42700)
0810 07:27:09.314924 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.314938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.314990 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.315352 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.315353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.315528 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.315598 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.315600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.317134 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.317136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.318002 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.318005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.318607 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.318608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.318675 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.319105 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.319110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.321255 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.322997 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.322999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.322999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.323226 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.323513 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.323514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.323515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.323583 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.323888 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.323888 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.323934 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.323980 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.324080 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 07:27:09.324563 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.314952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.24.60.252:46613)
0810 07:27:09.315147 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.315149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.315158 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:27:09.315181 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.315181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.315205 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.315678 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.315713 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.317027 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.318193 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.318501 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.318603 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.321353 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.321354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.321354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.321360 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.322903 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.323327 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.323328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.323329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.323418 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.323664 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.323665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.323666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.323809 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.324305 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.324347 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.324440 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":125,"threads_started":1}
I20260810 07:27:09.326040 8093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: Bootstrap starting.
I20260810 07:27:09.327293 8093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:09.327630 8093 log.cc:824] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:09.328536 8093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: No bootstrap required, opened a new log
I20260810 07:27:09.330142 8093 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.330288 8093 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:09.330339 8093 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ea0a3c0d23496994df531bf3bca82b, State: Initialized, Role: FOLLOWER
I20260810 07:27:09.330472 8093 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.330858 8219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.330940 8093 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:09.330945 8219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:27:09.333360 8231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:27:09.333421 8231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:27:09.333551 8011 master_runner.cc:186] Error getting master registration for 127.24.60.254:33935: OK, instance_id { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" instance_seqno: 1786346828899518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:27:09.336669 8011 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:27:09.349223 8004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } attempt: 3
Aug 10 07:27:09 dist-test-slave-vxj2 krb5kdc[7899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346828, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:27:09.358697 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.349626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:46613 (local address 127.0.0.1:42712)
0810 07:27:09.349686 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:09.349699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:09.349742 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:09.349944 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:09.349945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:09.350195 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:09.350258 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.350260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.351319 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.351320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.352176 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:09.352180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.352833 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.352834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:09.352905 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.353587 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:09.353593 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:09.355259 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:09.357394 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.357395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.357396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.357654 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.357898 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:09.357899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:09.357899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:09.357978 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:09.358328 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:09.358329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:09.358388 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 07:27:09.358466 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 07:27:09.358579 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 07:27:09.358712 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:09.349709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.24.60.252:46613)
0810 07:27:09.349772 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:09.349774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:09.349785 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:09.349825 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:09.349826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:09.349836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:09.349915 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:09.350351 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.351207 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.352393 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:09.352738 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:09.352812 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:09.355370 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:09.355372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:09.355372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:09.355382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:09.357288 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.357746 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.357747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.357747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.357819 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:09.358063 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:09.358064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:09.358065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:09.358242 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:09.358318 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:09.358513 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:27:09.358613 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:27:09.360926 8004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: Bootstrap starting.
I20260810 07:27:09.361863 8004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:09.362211 8004 log.cc:824] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:09.363265 8004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: No bootstrap required, opened a new log
I20260810 07:27:09.364426 8197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:09.364545 8197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.364840 8197 leader_election.cc:288] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96d92e561e574eeaaf7eb811228c80e4 (127.24.60.254:33935), e3ea0a3c0d23496994df531bf3bca82b (127.24.60.253:40557)
I20260810 07:27:09.365249 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ea0a3c0d23496994df531bf3bca82b" is_pre_election: true
I20260810 07:27:09.365442 8062 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80109bc4c6104a41bccdbc250cbe9ba4 in term 0.
I20260810 07:27:09.366060 8128 leader_election.cc:302] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80109bc4c6104a41bccdbc250cbe9ba4, e3ea0a3c0d23496994df531bf3bca82b; no voters:
I20260810 07:27:09.366216 8197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:09.366268 8197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:09.366300 8197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:09.367296 8197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.367458 8197 leader_election.cc:288] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [CANDIDATE]: Term 1 election: Requested vote from peers 96d92e561e574eeaaf7eb811228c80e4 (127.24.60.254:33935), e3ea0a3c0d23496994df531bf3bca82b (127.24.60.253:40557)
I20260810 07:27:09.367817 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ea0a3c0d23496994df531bf3bca82b"
I20260810 07:27:09.367918 8062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:09.368952 8062 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80109bc4c6104a41bccdbc250cbe9ba4 in term 1.
I20260810 07:27:09.369176 8128 leader_election.cc:302] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80109bc4c6104a41bccdbc250cbe9ba4, e3ea0a3c0d23496994df531bf3bca82b; no voters:
I20260810 07:27:09.369273 8197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:09.369462 8197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 80109bc4c6104a41bccdbc250cbe9ba4, State: Running, Role: LEADER
I20260810 07:27:09.369571 8197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.370034 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80109bc4c6104a41bccdbc250cbe9ba4. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.370369 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:09.370889 8004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.371055 8004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:09.371093 8004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d92e561e574eeaaf7eb811228c80e4, State: Initialized, Role: FOLLOWER
I20260810 07:27:09.371218 8004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } }
I20260810 07:27:09.371591 8004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:09.371677 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96d92e561e574eeaaf7eb811228c80e4"
I20260810 07:27:09.371802 7968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:09.372833 7968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80109bc4c6104a41bccdbc250cbe9ba4 in term 1.
I20260810 07:27:09.373291 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96d92e561e574eeaaf7eb811228c80e4" is_pre_election: true
I20260810 07:27:09.373399 7969 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 80109bc4c6104a41bccdbc250cbe9ba4 in term 1. Re-sending same reply.
I20260810 07:27:09.371613 8239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.374415 8239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:27:09.375259 8250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:27:09.375391 8250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:27:09.381467 8238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:09.381764 8238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:09.394137 8062 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [term 1 FOLLOWER]: Refusing update from remote peer 80109bc4c6104a41bccdbc250cbe9ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:09.394147 7969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [term 1 FOLLOWER]: Refusing update from remote peer 80109bc4c6104a41bccdbc250cbe9ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:09.394649 8235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:09.394953 8197 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:09.396296 8219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80109bc4c6104a41bccdbc250cbe9ba4. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.396418 8219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:27:09.397111 7943 tablet.cc:2369] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:09.398080 8239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80109bc4c6104a41bccdbc250cbe9ba4. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.398191 8239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:09.402613 8197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.402756 8197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:09.403594 8219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.403714 8219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:09.405417 8252 mvcc.cc:205] Tried to move back new op lower bound from 7316876613194866688 to 7316876613098389504. Current Snapshot: MvccSnapshot[applied={T|T < 7316876613194866688}]
I20260810 07:27:09.406085 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.406188 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:09.407017 8239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d92e561e574eeaaf7eb811228c80e4" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33935 } } peers { permanent_uuid: "e3ea0a3c0d23496994df531bf3bca82b" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40557 } } peers { permanent_uuid: "80109bc4c6104a41bccdbc250cbe9ba4" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 46613 } } }
I20260810 07:27:09.407126 8239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:09.407490 8255 mvcc.cc:205] Tried to move back new op lower bound from 7316876613194866688 to 7316876613098389504. Current Snapshot: MvccSnapshot[applied={T|T < 7316876613194866688}]
I20260810 07:27:09.407794 8253 mvcc.cc:205] Tried to move back new op lower bound from 7316876613194866688 to 7316876613098389504. Current Snapshot: MvccSnapshot[applied={T|T < 7316876613194866688}]
I20260810 07:27:09.408717 8238 catalog_manager.cc:1347] Generated new cluster ID: 22a38e7d316c4c2988296d37e2848a2f
I20260810 07:27:09.408797 8238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:09.430003 8238 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:09.430948 8238 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:09.441715 8238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4: Generated new TSK 0
I20260810 07:27:09.441987 8238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:27:10.333968 8231 catalog_manager.cc:1259] Loaded cluster ID: 22a38e7d316c4c2988296d37e2848a2f
I20260810 07:27:10.334059 8231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: loading cluster ID for follower catalog manager: success
I20260810 07:27:10.334952 8231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: acquiring CA information for follower catalog manager: success
I20260810 07:27:10.335295 8231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3ea0a3c0d23496994df531bf3bca82b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:27:10.375963 8250 catalog_manager.cc:1259] Loaded cluster ID: 22a38e7d316c4c2988296d37e2848a2f
I20260810 07:27:10.376057 8250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: loading cluster ID for follower catalog manager: success
I20260810 07:27:10.377024 8250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: acquiring CA information for follower catalog manager: success
I20260810 07:27:10.377400 8250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96d92e561e574eeaaf7eb811228c80e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:27:10.508471 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 7917
W20260810 07:27:10.516981 8130 connection.cc:535] client connection to 127.24.60.254:33935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 07:27:10.517107 8130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:27:10.517274 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8011
I20260810 07:27:10.520370 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:10.519888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33935 (local address 127.0.0.1:44494)
0810 07:27:10.520260 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:10.520318 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:33935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":295,"threads_started":1}
W20260810 07:27:10.520462 8130 consensus_peers.cc:487] T 00000000000000000000000000000000 P 80109bc4c6104a41bccdbc250cbe9ba4 -> Peer 96d92e561e574eeaaf7eb811228c80e4 (127.24.60.254:33935): Couldn't send request to peer 96d92e561e574eeaaf7eb811228c80e4. Status: Network error: Client connection negotiation failed: client connection to 127.24.60.254:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 07:27:10.526137 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8101
2026-08-10T07:27:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3032 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:10 dist-test-slave-vxj2 krb5kdc[8271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:10 dist-test-slave-vxj2 krb5kdc[8271](info): set up 2 sockets
Aug 10 07:27:10 dist-test-slave-vxj2 krb5kdc[8271](info): commencing operation
krb5kdc: starting...
W20260810 07:27:11.693079 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:27:11 dist-test-slave-vxj2 krb5kdc[8271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:27:11.829227 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35979
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:35979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:11.915882 8292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:11.916167 8292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:11.916211 8292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:11.917572 8292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:11.917639 8292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:11.917661 8292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:11.917694 8292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:11.917721 8292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:11.919329 8292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:35979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:11.919633 8292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:11.919869 8292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:11.925263 8298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:11.925460 8292 server_base.cc:1034] running on GCE node
W20260810 07:27:11.925254 8302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:11.925254 8299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:11.925803 8292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:11.926137 8292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:11.927363 8292 hybrid_clock.cc:648] HybridClock initialized: now 1786346831927340 us; error 40 us; skew 500 ppm
Aug 10 07:27:11 dist-test-slave-vxj2 krb5kdc[8271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346831, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:11.934383 8292 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:11.934978 8292 webserver.cc:460] Webserver started at http://127.24.60.254:42423/ using document root <none> and password file <none>
I20260810 07:27:11.935225 8292 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:11.935297 8292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:11.935459 8292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:11.936643 8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b"
format_stamp: "Formatted at 2026-08-10 07:27:11 on dist-test-slave-vxj2"
server_key: "d7c29e3be32aa6477ecd7c0639372227"
server_key_iv: "1d61a98bbd81952f952409b8979b41fa"
server_key_version: "encryptionkey@0"
I20260810 07:27:11.937040 8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b"
format_stamp: "Formatted at 2026-08-10 07:27:11 on dist-test-slave-vxj2"
server_key: "d7c29e3be32aa6477ecd7c0639372227"
server_key_iv: "1d61a98bbd81952f952409b8979b41fa"
server_key_version: "encryptionkey@0"
I20260810 07:27:11.938627 8292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:11.939581 8309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:11.939852 8292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:11.939997 8292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b"
format_stamp: "Formatted at 2026-08-10 07:27:11 on dist-test-slave-vxj2"
server_key: "d7c29e3be32aa6477ecd7c0639372227"
server_key_iv: "1d61a98bbd81952f952409b8979b41fa"
server_key_version: "encryptionkey@0"
I20260810 07:27:11.940088 8292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:11.968436 8292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:11.971187 8292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:11.971395 8292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:11.976024 8292 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:35979
I20260810 07:27:11.976085 8371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:35979 every 8 connection(s)
I20260810 07:27:11.976488 8292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:11.977428 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:11.979899 8372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b: Bootstrap starting.
I20260810 07:27:11.980554 8372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:11.980876 8372 log.cc:824] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:11.981714 8372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b: No bootstrap required, opened a new log
I20260810 07:27:11.983448 8372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35979 } }
I20260810 07:27:11.983599 8372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:11.983649 8372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bfd12ce5004ffbad5e1eb0bf1aec7b, State: Initialized, Role: FOLLOWER
I20260810 07:27:11.983819 8372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35979 } }
I20260810 07:27:11.983886 8372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:11.983909 8372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:11.983971 8372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:11.984566 8372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35979 } }
I20260810 07:27:11.984683 8372 leader_election.cc:302] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67bfd12ce5004ffbad5e1eb0bf1aec7b; no voters:
I20260810 07:27:11.984889 8372 leader_election.cc:288] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:11.984961 8375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:11.985263 8375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [term 1 LEADER]: Becoming Leader. State: Replica: 67bfd12ce5004ffbad5e1eb0bf1aec7b, State: Running, Role: LEADER
I20260810 07:27:11.985320 8372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:11.985409 8375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35979 } }
I20260810 07:27:11.985738 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8292
I20260810 07:27:11.985852 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:27:11.986089 24819 external_mini_cluster.cc:1428] Setting key fde8b411c9008c6d54e7562c131d080d
I20260810 07:27:11.986045 8377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35979 } } }
I20260810 07:27:11.986135 8377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:11.986213 8378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67bfd12ce5004ffbad5e1eb0bf1aec7b. Latest consensus state: current_term: 1 leader_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bfd12ce5004ffbad5e1eb0bf1aec7b" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35979 } } }
I20260810 07:27:11.986310 8378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:11.986601 8386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:11.987455 8386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:11.989609 8386 catalog_manager.cc:1347] Generated new cluster ID: d081136240d349b1bed5dfc70ab2a824
I20260810 07:27:11.989670 8386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:27:11 dist-test-slave-vxj2 krb5kdc[8271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:11.999035 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:11.986821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.24.60.254:35979)
0810 07:27:11.988168 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:11.988179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:11.988223 (+ 44us) server_negotiation.cc:407] Connection header received
0810 07:27:11.988384 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:11.988392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:11.988452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:11.988605 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:11.989593 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:11.990696 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:11.991571 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:11.992091 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:11.992244 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:11.995067 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:11.995073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:11.995073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:11.995090 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:11.997105 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:11.997636 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:11.997640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:11.997640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:11.997765 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:11.998026 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:11.998027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:11.998027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:11.998328 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:11.998516 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:11.998640 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 07:27:11.998823 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":67,"threads_started":1}
I20260810 07:27:12.000195 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:27:12.006245 8386 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:12.006868 8386 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:12.027894 8386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67bfd12ce5004ffbad5e1eb0bf1aec7b: Generated new TSK 0
I20260810 07:27:12.028188 8386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:27:12.054430 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8292
2026-08-10T07:27:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1526 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:12 dist-test-slave-vxj2 krb5kdc[8401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:12 dist-test-slave-vxj2 krb5kdc[8401](info): set up 2 sockets
Aug 10 07:27:12 dist-test-slave-vxj2 krb5kdc[8401](info): commencing operation
krb5kdc: starting...
W20260810 07:27:13.189122 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:27:13.328755 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42767
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:13.421028 8417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:13.421437 8417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:13.421489 8417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:13.422899 8417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:13.422987 8417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:13.423010 8417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:13.423033 8417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:13.423061 8417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:13.424813 8417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:42767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:13.425184 8417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:13.425472 8417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:13.431133 8429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.431159 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.431208 8425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:13.431308 8417 server_base.cc:1034] running on GCE node
I20260810 07:27:13.431695 8417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:13.432045 8417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:13.433226 8417 hybrid_clock.cc:648] HybridClock initialized: now 1786346833433176 us; error 62 us; skew 500 ppm
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:13.436051 8417 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:13.436614 8417 webserver.cc:460] Webserver started at http://127.24.60.254:34259/ using document root <none> and password file <none>
I20260810 07:27:13.436875 8417 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:13.436956 8417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:13.437116 8417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:13.438459 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "db9f983567e64c2db9ea68bdd12e5551"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "23c9d91589a2cd7262880ea14a7c92d1"
server_key_iv: "30527c592f7d602acf3774720751f4bc"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.438851 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "db9f983567e64c2db9ea68bdd12e5551"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "23c9d91589a2cd7262880ea14a7c92d1"
server_key_iv: "30527c592f7d602acf3774720751f4bc"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.440302 8417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:27:13.441222 8437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.441529 8417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:13.441692 8417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "db9f983567e64c2db9ea68bdd12e5551"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "23c9d91589a2cd7262880ea14a7c92d1"
server_key_iv: "30527c592f7d602acf3774720751f4bc"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.441784 8417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:13.460153 8417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:13.462625 8417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:13.462837 8417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:13.467412 8417 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:42767
I20260810 07:27:13.467546 8505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:42767 every 8 connection(s)
I20260810 07:27:13.467907 8417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:13.468964 8507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:13.471640 8507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap starting.
I20260810 07:27:13.472364 8507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:13.472715 8507 log.cc:824] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:13.473850 8507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: No bootstrap required, opened a new log
I20260810 07:27:13.475195 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8417
I20260810 07:27:13.475361 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:27:13.475586 24819 external_mini_cluster.cc:1428] Setting key 09e3f33fa388e75848a2248b6056b8fb
I20260810 07:27:13.476445 8507 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:13.476651 8507 raft_consensus.cc:374] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:13.476677 8507 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9f983567e64c2db9ea68bdd12e5551, State: Initialized, Role: FOLLOWER
I20260810 07:27:13.476833 8507 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:13.476899 8507 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:13.476922 8507 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:13.476997 8507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:13.478360 8507 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:13.478468 8507 leader_election.cc:302] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9f983567e64c2db9ea68bdd12e5551; no voters:
I20260810 07:27:13.478696 8507 leader_election.cc:288] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:13.478768 8513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:13.478955 8513 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 LEADER]: Becoming Leader. State: Replica: db9f983567e64c2db9ea68bdd12e5551, State: Running, Role: LEADER
I20260810 07:27:13.479065 8507 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:13.479053 8513 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:13.479452 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db9f983567e64c2db9ea68bdd12e5551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } } }
I20260810 07:27:13.479491 8515 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9f983567e64c2db9ea68bdd12e5551. Latest consensus state: current_term: 1 leader_uuid: "db9f983567e64c2db9ea68bdd12e5551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } } }
I20260810 07:27:13.479552 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:13.479574 8515 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:13.480607 8522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:13.481293 8522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:13.483239 8522 catalog_manager.cc:1347] Generated new cluster ID: 5cf2e7917e784b659b26a81c803cbb08
I20260810 07:27:13.483295 8522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:13.489032 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:13.476326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.24.60.254:42767)
0810 07:27:13.476544 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:13.476553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:13.476580 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:27:13.476724 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:13.476732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:13.476816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:13.477003 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:13.477993 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.479279 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.480384 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.481058 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.481211 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:13.484420 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:13.484425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:13.484426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:13.484440 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:13.486697 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:13.487215 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:13.487219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:13.487219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:13.487370 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:13.487627 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:13.487628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:13.487628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:13.487953 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:13.488582 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:13.488656 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 07:27:13.488811 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":107,"threads_started":1}
I20260810 07:27:13.491395 8522 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:13.491977 8522 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:13.498302 8522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Generated new TSK 0
I20260810 07:27:13.498572 8522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:27:13.536923 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42767
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:13.630822 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:13.631063 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:13.631091 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:13.632418 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:13.632477 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:13.632506 8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:13.634282 8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:13.634595 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:13.634877 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:13.640414 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.640422 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.640422 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:13.640781 8540 server_base.cc:1034] running on GCE node
I20260810 07:27:13.641000 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:13.641286 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:13.642481 8540 hybrid_clock.cc:648] HybridClock initialized: now 1786346833642472 us; error 43 us; skew 500 ppm
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:13.645431 8540 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:13.645970 8540 webserver.cc:460] Webserver started at http://127.24.60.193:37233/ using document root <none> and password file <none>
I20260810 07:27:13.646231 8540 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:13.646303 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:13.646469 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:13.647588 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "fb657e7c7e6a4c58b4be58e80416c021"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "749ea85489ba1e5175f51793ed228bd1"
server_key_iv: "70f557b393e29ca6091b6441649b7917"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.647969 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "fb657e7c7e6a4c58b4be58e80416c021"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "749ea85489ba1e5175f51793ed228bd1"
server_key_iv: "70f557b393e29ca6091b6441649b7917"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.649588 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:27:13.650455 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.650698 8540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:13.650826 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "fb657e7c7e6a4c58b4be58e80416c021"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "749ea85489ba1e5175f51793ed228bd1"
server_key_iv: "70f557b393e29ca6091b6441649b7917"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.650939 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:13.671108 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:13.673434 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:13.673619 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:13.673902 8540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:13.674253 8540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:13.674289 8540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.674314 8540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:13.674364 8540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.680977 8540 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:40379
I20260810 07:27:13.681084 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:40379 every 8 connection(s)
I20260810 07:27:13.681490 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:13.684013 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8540
I20260810 07:27:13.684161 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:27:13.684377 24819 external_mini_cluster.cc:1428] Setting key 5eb4827ea390347b5fdf3db9c708a1fb
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:13.692744 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:13.682413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:40567 (local address 127.24.60.254:42767)
0810 07:27:13.682504 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:13.682506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:13.682733 (+ 227us) server_negotiation.cc:407] Connection header received
0810 07:27:13.683266 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:13.683268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:13.683281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:13.683346 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:13.684441 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.685026 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.686095 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.686452 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.686543 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:13.689362 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:13.689364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:13.689364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:13.689372 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:13.691022 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:13.691580 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:13.691581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:13.691582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:13.691654 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:13.691977 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:13.691978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:13.691979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:13.692123 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:13.692211 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:13.692537 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 07:27:13.692641 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:27:13.692965 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:13.682386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:40567)
0810 07:27:13.682582 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:13.682630 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:13.683139 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:13.683412 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:13.683416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:13.684026 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:13.684330 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:13.684337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.685114 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.685115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:13.685998 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:13.686000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.686592 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.686593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:13.686766 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:13.687512 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:13.687524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:13.689253 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:13.691157 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:13.691167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:13.691169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:13.691468 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:13.691733 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:13.691734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:13.691734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:13.691893 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:13.692201 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:13.692209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:13.692377 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 07:27:13.692504 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 07:27:13.692691 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
I20260810 07:27:13.693711 8690 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42767
I20260810 07:27:13.693873 8690 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:13.694118 8690 heartbeater.cc:511] Master 127.24.60.254:42767 requested a full tablet report, sending...
I20260810 07:27:13.694723 8460 ts_manager.cc:194] Registered new tserver with Master: fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379)
I20260810 07:27:13.695421 8460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:40567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:27:13.716117 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42767
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:13.803145 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:13.803377 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:13.803406 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:13.804529 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:13.804567 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:13.804592 8697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:13.806042 8697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:13.806248 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:13.806447 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:13.811793 8706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.811805 8703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.811793 8704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:13.812034 8697 server_base.cc:1034] running on GCE node
I20260810 07:27:13.812240 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:13.812515 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:13.813689 8697 hybrid_clock.cc:648] HybridClock initialized: now 1786346833813676 us; error 39 us; skew 500 ppm
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:13.816829 8697 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:13.817440 8697 webserver.cc:460] Webserver started at http://127.24.60.194:36549/ using document root <none> and password file <none>
I20260810 07:27:13.817690 8697 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:13.817759 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:13.817907 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:13.819010 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "66da1aded80947009bcdb49c97346cc3"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "ec7728f348ee570df16f1368fac79246"
server_key_iv: "e8f8bf82d35b3282eafb7edfc7c8f8a4"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.819401 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "66da1aded80947009bcdb49c97346cc3"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "ec7728f348ee570df16f1368fac79246"
server_key_iv: "e8f8bf82d35b3282eafb7edfc7c8f8a4"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.820909 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:27:13.821805 8714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.821995 8697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:13.822098 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "66da1aded80947009bcdb49c97346cc3"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "ec7728f348ee570df16f1368fac79246"
server_key_iv: "e8f8bf82d35b3282eafb7edfc7c8f8a4"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.822180 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:13.845685 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:13.848517 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:13.848877 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:13.849200 8697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:13.849620 8697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:13.849679 8697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.849722 8697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:13.849745 8697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:13.856222 8697 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:44699
I20260810 07:27:13.856288 8855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:44699 every 8 connection(s)
I20260810 07:27:13.856683 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:13.862360 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8697
I20260810 07:27:13.862461 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:27:13.862671 24819 external_mini_cluster.cc:1428] Setting key c65d02d962c47d27db453942d0edb86c
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:13.868273 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:13.857632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:37285 (local address 127.24.60.254:42767)
0810 07:27:13.857710 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:13.857713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:13.858002 (+ 289us) server_negotiation.cc:407] Connection header received
0810 07:27:13.858544 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:13.858545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:13.858557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:13.858626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:13.859725 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.860314 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.861526 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.861828 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.861915 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:13.864781 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:13.864783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:13.864784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:13.864793 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:13.866520 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:13.867106 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:13.867107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:13.867107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:13.867173 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:13.867505 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:13.867505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:13.867506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:13.867624 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:13.867678 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:13.868107 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 07:27:13.868201 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:27:13.868585 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:13.857611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:37285)
0810 07:27:13.857848 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:13.857893 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:13.858426 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:13.858688 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:13.858692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:13.859400 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:13.859617 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:13.859623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.860456 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.860457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:13.861386 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:13.861390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:13.862002 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:13.862003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:13.862132 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:13.862923 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:13.862932 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:13.864671 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:13.866623 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:13.866634 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:13.866637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:13.867011 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:13.867248 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:13.867249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:13.867249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:13.867427 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:13.867703 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:13.867711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:13.867940 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 07:27:13.868082 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 07:27:13.868295 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260810 07:27:13.869506 8856 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42767
I20260810 07:27:13.869642 8856 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:13.869853 8856 heartbeater.cc:511] Master 127.24.60.254:42767 requested a full tablet report, sending...
I20260810 07:27:13.870327 8460 ts_manager.cc:194] Registered new tserver with Master: 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
I20260810 07:27:13.870898 8460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:37285
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:27:13.894176 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42767
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:13.982494 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:13.982744 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:13.982772 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:13.984053 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:13.984113 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:13.984148 8862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:13.985845 8862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:13.986142 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:13.986409 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:13.992090 8868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.992106 8869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:13.992152 8871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:13.992376 8862 server_base.cc:1034] running on GCE node
I20260810 07:27:13.992622 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:13.992839 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:13.994014 8862 hybrid_clock.cc:648] HybridClock initialized: now 1786346833994006 us; error 39 us; skew 500 ppm
Aug 10 07:27:13 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:13.996790 8862 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:13.997377 8862 webserver.cc:460] Webserver started at http://127.24.60.195:41685/ using document root <none> and password file <none>
I20260810 07:27:13.997635 8862 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:13.997706 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:13.997859 8862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:13.999033 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "1f03f53f7e4349588b16d484bbb9e436"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "7964412ba16fd1eed016a2c4fcd9034a"
server_key_iv: "9d0f00f63cd205e3f7e7309cd00768f3"
server_key_version: "encryptionkey@0"
I20260810 07:27:13.999462 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "1f03f53f7e4349588b16d484bbb9e436"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "7964412ba16fd1eed016a2c4fcd9034a"
server_key_iv: "9d0f00f63cd205e3f7e7309cd00768f3"
server_key_version: "encryptionkey@0"
I20260810 07:27:14.000891 8862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:27:14.001855 8879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.002053 8862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:14.002156 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "1f03f53f7e4349588b16d484bbb9e436"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "7964412ba16fd1eed016a2c4fcd9034a"
server_key_iv: "9d0f00f63cd205e3f7e7309cd00768f3"
server_key_version: "encryptionkey@0"
I20260810 07:27:14.002242 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:14.014140 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:14.016736 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:14.016925 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:14.017295 8862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:14.017652 8862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:14.017730 8862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.017774 8862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:14.017800 8862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.024204 8862 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:36953
I20260810 07:27:14.024248 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:36953 every 8 connection(s)
I20260810 07:27:14.024595 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:14.030267 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 8862
I20260810 07:27:14.030366 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:27:14.030591 24819 external_mini_cluster.cc:1428] Setting key 534e6b018b45fbc4fa3c88eed6f32960
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346833, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:14.034925 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.025478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:49493 (local address 127.24.60.254:42767)
0810 07:27:14.025550 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.025553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.025777 (+ 224us) server_negotiation.cc:407] Connection header received
0810 07:27:14.026391 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.026393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.026407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.026477 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:14.027523 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.028023 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.029003 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.029270 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.029329 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.031885 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:14.031887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:14.031887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:14.031894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:14.033511 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.033989 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.033990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.033991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.034055 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.034303 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.034304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.034304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.034421 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:14.034475 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.034725 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.034805 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:27:14.035075 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.025444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:49493)
0810 07:27:14.025637 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.025685 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.026269 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.026565 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.026569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.027179 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:14.027392 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.027397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.028112 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.028113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.028917 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.028919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.029337 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.029338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.029450 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.030001 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:14.030010 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:14.031756 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:14.033608 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.033616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.033618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.033891 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.034130 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.034130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.034131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.034245 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.034478 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:14.034482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:14.034611 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.034698 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.034850 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260810 07:27:14.035583 9017 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42767
I20260810 07:27:14.035712 9017 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:14.035940 9017 heartbeater.cc:511] Master 127.24.60.254:42767 requested a full tablet report, sending...
I20260810 07:27:14.036357 8460 ts_manager.cc:194] Registered new tserver with Master: 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953)
I20260810 07:27:14.036767 8460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:49493
I20260810 07:27:14.043200 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:14.052275 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.044836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.24.60.254:42767)
0810 07:27:14.044905 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.044907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.045008 (+ 101us) server_negotiation.cc:407] Connection header received
0810 07:27:14.045069 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.045069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.045078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.045207 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:14.045928 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.046432 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.047519 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.047729 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.047832 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.049323 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:14.049325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:14.049325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:14.049333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:14.050978 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.051382 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.051384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.051384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.051449 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.051666 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.051667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.051668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.051847 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:14.051959 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.052072 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.052163 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:27:14.054502 8460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:14.056226 8460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:14.063575 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.059010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.24.60.194:44699)
0810 07:27:14.059541 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.059547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.059558 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:14.059625 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.059627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.059645 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.059960 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:14.060051 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.061219 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.062560 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.063081 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.063180 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.063250 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.063424 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.063470 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":81,"threads_started":1}
I20260810 07:27:14.063634 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.059067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36953 (local address 127.0.0.1:37698)
0810 07:27:14.059176 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.059187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.059229 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.059885 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.059886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.059887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:14.059937 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.059939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.060901 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.060902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.062247 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.062251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.063223 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.063224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.063307 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.063313 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.063466 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.063519 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260810 07:27:14.063616 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.058784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44699 (local address 127.0.0.1:53598)
0810 07:27:14.058919 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.058965 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.059039 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.059761 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.059765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.059767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:14.059871 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.059873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.061330 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.061331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.062469 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.062471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.063181 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.063182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.063276 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.063291 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.063403 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.063468 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":85,"thread_start_us":165,"threads_started":1}
I20260810 07:27:14.063832 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.058802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:40379 (local address 127.0.0.1:54446)
0810 07:27:14.058999 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.059010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.059074 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.059863 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.059864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.059874 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:14.059960 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.059963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.061345 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.061346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.062288 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.062291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.063460 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.063461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.063524 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.063528 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.063582 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.063623 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
I20260810 07:27:14.063843 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.059222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.24.60.193:40379)
0810 07:27:14.059569 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.059574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.059591 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:27:14.059648 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.059651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.059668 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.059765 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:14.060050 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.061220 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.062426 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.063183 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.063243 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.063305 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.063679 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.063735 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
I20260810 07:27:14.064625 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.059314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.24.60.195:36953)
0810 07:27:14.059708 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.059713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.059722 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:27:14.059766 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.059768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.059784 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.060020 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:14.060099 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.060787 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.062356 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.063106 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.064345 (+ 1239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.064451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.064480 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.064536 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":106,"threads_started":1}
I20260810 07:27:14.064646 8610 tablet_service.cc:1467] Processing CreateTablet for tablet af5e91a5b6504848b250d2097eecee9b (DEFAULT_TABLE table=test-table [id=0f0a20868c544d698914a3457b16569b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:14.065058 8610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5e91a5b6504848b250d2097eecee9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:14.065044 8774 tablet_service.cc:1467] Processing CreateTablet for tablet af5e91a5b6504848b250d2097eecee9b (DEFAULT_TABLE table=test-table [id=0f0a20868c544d698914a3457b16569b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:14.065299 8936 tablet_service.cc:1467] Processing CreateTablet for tablet af5e91a5b6504848b250d2097eecee9b (DEFAULT_TABLE table=test-table [id=0f0a20868c544d698914a3457b16569b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:14.065415 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5e91a5b6504848b250d2097eecee9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:14.065635 8936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5e91a5b6504848b250d2097eecee9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:14.068442 9035 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Bootstrap starting.
I20260810 07:27:14.068928 9037 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Bootstrap starting.
I20260810 07:27:14.068959 9038 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap starting.
I20260810 07:27:14.069298 9035 tablet_bootstrap.cc:654] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:14.069687 9037 tablet_bootstrap.cc:654] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:14.069689 9038 tablet_bootstrap.cc:654] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:14.069926 9035 log.cc:824] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:14.070365 9038 log.cc:824] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:14.070387 9037 log.cc:824] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:14.071076 9035 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: No bootstrap required, opened a new log
I20260810 07:27:14.071166 9035 ts_tablet_manager.cc:1397] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:14.071271 9038 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: No bootstrap required, opened a new log
I20260810 07:27:14.071352 9038 ts_tablet_manager.cc:1397] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:14.073556 9037 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: No bootstrap required, opened a new log
I20260810 07:27:14.073664 9037 ts_tablet_manager.cc:1397] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 07:27:14.073742 9035 raft_consensus.cc:348] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.073742 9038 raft_consensus.cc:348] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.073956 9035 raft_consensus.cc:374] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:14.073956 9038 raft_consensus.cc:374] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:14.074012 9035 raft_consensus.cc:729] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f53f7e4349588b16d484bbb9e436, State: Initialized, Role: FOLLOWER
I20260810 07:27:14.074012 9038 raft_consensus.cc:729] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb657e7c7e6a4c58b4be58e80416c021, State: Initialized, Role: FOLLOWER
I20260810 07:27:14.074152 9038 consensus_queue.cc:260] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.074199 9035 consensus_queue.cc:260] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.074508 9035 ts_tablet_manager.cc:1428] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:14.074508 9038 ts_tablet_manager.cc:1428] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 07:27:14.074659 8690 heartbeater.cc:503] Master 127.24.60.254:42767 was elected leader, sending a full tablet report...
I20260810 07:27:14.074700 9017 heartbeater.cc:503] Master 127.24.60.254:42767 was elected leader, sending a full tablet report...
I20260810 07:27:14.076066 9037 raft_consensus.cc:348] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.076282 9037 raft_consensus.cc:374] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:14.076331 9037 raft_consensus.cc:729] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66da1aded80947009bcdb49c97346cc3, State: Initialized, Role: FOLLOWER
I20260810 07:27:14.076463 9037 consensus_queue.cc:260] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.076920 8856 heartbeater.cc:503] Master 127.24.60.254:42767 was elected leader, sending a full tablet report...
I20260810 07:27:14.076984 9037 ts_tablet_manager.cc:1428] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260810 07:27:14.107375 8857 tablet.cc:2369] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:27:14.182379 8691 tablet.cc:2369] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:27:14.275301 9018 tablet.cc:2369] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:14.305178 9043 raft_consensus.cc:482] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:14.305348 9043 raft_consensus.cc:504] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.305950 9043 leader_election.cc:288] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699), fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379)
I20260810 07:27:14.309440 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.306180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:40379 (local address 127.24.60.195:32829)
0810 07:27:14.306361 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.306379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.306457 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.306676 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.306676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.306678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:14.306771 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.306772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.307485 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.307486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.308408 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.308411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.309101 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.309102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.309261 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.309267 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.309330 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.309366 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":87,"thread_start_us":106,"threads_started":1}
I20260810 07:27:14.309501 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.306276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:32829 (local address 127.24.60.193:40379)
0810 07:27:14.306331 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.306333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.306533 (+ 200us) server_negotiation.cc:407] Connection header received
0810 07:27:14.306566 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.306567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.306583 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.306660 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:14.306871 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.307350 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.308621 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.308995 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.309066 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.309074 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.309363 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.309413 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 07:27:14.309643 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.306241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:32909 (local address 127.24.60.194:44699)
0810 07:27:14.306296 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.306298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.306312 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:27:14.306410 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.306411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.306424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.306499 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:14.306745 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.307321 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.308561 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.309140 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.309245 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.309256 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.309503 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.309554 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 07:27:14.309594 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.306166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44699 (local address 127.24.60.195:32909)
0810 07:27:14.306242 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.306257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.306313 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.306558 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.306559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.306562 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:14.306656 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.306658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.307438 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.307439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.308377 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.308388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.309263 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.309264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.309405 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.309409 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.309472 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.309518 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":42}
I20260810 07:27:14.310050 8634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5e91a5b6504848b250d2097eecee9b" candidate_uuid: "1f03f53f7e4349588b16d484bbb9e436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb657e7c7e6a4c58b4be58e80416c021" is_pre_election: true
I20260810 07:27:14.310053 8795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5e91a5b6504848b250d2097eecee9b" candidate_uuid: "1f03f53f7e4349588b16d484bbb9e436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66da1aded80947009bcdb49c97346cc3" is_pre_election: true
I20260810 07:27:14.310230 8795 raft_consensus.cc:2454] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f53f7e4349588b16d484bbb9e436 in term 0.
I20260810 07:27:14.310230 8634 raft_consensus.cc:2454] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f53f7e4349588b16d484bbb9e436 in term 0.
I20260810 07:27:14.310505 8882 leader_election.cc:302] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f53f7e4349588b16d484bbb9e436, fb657e7c7e6a4c58b4be58e80416c021; no voters:
I20260810 07:27:14.310673 9043 raft_consensus.cc:2781] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:14.310758 9043 raft_consensus.cc:482] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:14.310789 9043 raft_consensus.cc:3037] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:14.311581 9043 raft_consensus.cc:504] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.311748 9043 leader_election.cc:288] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [CANDIDATE]: Term 1 election: Requested vote from peers 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699), fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379)
I20260810 07:27:14.311976 8634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5e91a5b6504848b250d2097eecee9b" candidate_uuid: "1f03f53f7e4349588b16d484bbb9e436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb657e7c7e6a4c58b4be58e80416c021"
I20260810 07:27:14.311976 8795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5e91a5b6504848b250d2097eecee9b" candidate_uuid: "1f03f53f7e4349588b16d484bbb9e436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66da1aded80947009bcdb49c97346cc3"
I20260810 07:27:14.312072 8795 raft_consensus.cc:3037] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:14.312072 8634 raft_consensus.cc:3037] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:14.313057 8795 raft_consensus.cc:2454] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f53f7e4349588b16d484bbb9e436 in term 1.
I20260810 07:27:14.313092 8634 raft_consensus.cc:2454] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f53f7e4349588b16d484bbb9e436 in term 1.
I20260810 07:27:14.313344 8882 leader_election.cc:302] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f53f7e4349588b16d484bbb9e436, fb657e7c7e6a4c58b4be58e80416c021; no voters:
I20260810 07:27:14.313481 9043 raft_consensus.cc:2781] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:14.313725 9043 raft_consensus.cc:686] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 1 LEADER]: Becoming Leader. State: Replica: 1f03f53f7e4349588b16d484bbb9e436, State: Running, Role: LEADER
I20260810 07:27:14.313874 9043 consensus_queue.cc:237] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.314868 8459 catalog_manager.cc:5495] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195). New cstate: current_term: 1 leader_uuid: "1f03f53f7e4349588b16d484bbb9e436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } health_report { overall_health: HEALTHY } } }
I20260810 07:27:14.352368 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.349729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.24.60.195:36953)
0810 07:27:14.349804 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.349806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.349814 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:27:14.349857 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.349858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.349879 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.349969 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:14.350133 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.350622 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.351513 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.351718 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.351802 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.351985 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:14.352100 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:14.352199 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.352217 (+ 18us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.352240 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:27:14.354766 8634 raft_consensus.cc:1261] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Refusing update from remote peer 1f03f53f7e4349588b16d484bbb9e436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:14.354766 8795 raft_consensus.cc:1261] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Refusing update from remote peer 1f03f53f7e4349588b16d484bbb9e436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:14.355221 9043 consensus_queue.cc:1035] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:14.355391 9046 consensus_queue.cc:1035] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:14.357978 9049 mvcc.cc:205] Tried to move back new op lower bound from 7316876633513558016 to 7316876633350324224. Current Snapshot: MvccSnapshot[applied={T|T < 7316876633513558016}]
I20260810 07:27:14.358662 9050 mvcc.cc:205] Tried to move back new op lower bound from 7316876633513558016 to 7316876633350324224. Current Snapshot: MvccSnapshot[applied={T|T < 7316876633513558016}]
I20260810 07:27:14.361135 8459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:14.361306 8459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:14.362207 8459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Sending DeleteTablet for 3 replicas of tablet af5e91a5b6504848b250d2097eecee9b
I20260810 07:27:14.362594 8610 tablet_service.cc:1514] Processing DeleteTablet for tablet af5e91a5b6504848b250d2097eecee9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:14 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:54446
I20260810 07:27:14.362605 8936 tablet_service.cc:1514] Processing DeleteTablet for tablet af5e91a5b6504848b250d2097eecee9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:14 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:37698
I20260810 07:27:14.362898 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet af5e91a5b6504848b250d2097eecee9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:14 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:53598
I20260810 07:27:14.362979 9060 tablet_replica.cc:330] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: stopping tablet replica
I20260810 07:27:14.362984 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8540
I20260810 07:27:14.363096 9060 raft_consensus.cc:2229] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:14.363214 9060 raft_consensus.cc:2258] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:14.367998 9062 tablet_replica.cc:330] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: stopping tablet replica
I20260810 07:27:14.368186 9062 raft_consensus.cc:2229] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:14.368321 9061 tablet_replica.cc:330] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: stopping tablet replica
I20260810 07:27:14.368352 9062 raft_consensus.cc:2258] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:14.368438 9061 raft_consensus.cc:2229] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:14.368580 9061 raft_consensus.cc:2258] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:14.369421 9061 ts_tablet_manager.cc:1905] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:14.369421 9062 ts_tablet_manager.cc:1905] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:14.371304 9061 ts_tablet_manager.cc:1918] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:14.371405 9061 log.cc:1196] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/af5e91a5b6504848b250d2097eecee9b
I20260810 07:27:14.371438 9062 ts_tablet_manager.cc:1918] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:14.371510 9062 log.cc:1196] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/af5e91a5b6504848b250d2097eecee9b
I20260810 07:27:14.371722 9061 ts_tablet_manager.cc:1939] T af5e91a5b6504848b250d2097eecee9b P 1f03f53f7e4349588b16d484bbb9e436: Deleting consensus metadata
I20260810 07:27:14.371773 9062 ts_tablet_manager.cc:1939] T af5e91a5b6504848b250d2097eecee9b P 66da1aded80947009bcdb49c97346cc3: Deleting consensus metadata
I20260810 07:27:14.372237 8439 catalog_manager.cc:4841] TS 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953): tablet af5e91a5b6504848b250d2097eecee9b (table test-table [id=0f0a20868c544d698914a3457b16569b]) successfully deleted
I20260810 07:27:14.372553 8439 catalog_manager.cc:4841] TS 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699): tablet af5e91a5b6504848b250d2097eecee9b (table test-table [id=0f0a20868c544d698914a3457b16569b]) successfully deleted
W20260810 07:27:14.375087 8440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:40379 (error 108)
I20260810 07:27:14.375222 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8697
I20260810 07:27:14.379498 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.379236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:40379 (local address 127.0.0.1:53942)
0810 07:27:14.379330 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.379397 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:40379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 07:27:14.379777 8440 catalog_manager.cc:4584] TS fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af5e91a5b6504848b250d2097eecee9b: Network error: Client connection negotiation failed: client connection to 127.24.60.193:40379: connect: Connection refused (error 111)
I20260810 07:27:14.383185 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8862
I20260810 07:27:14.394722 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 8417
I20260810 07:27:14.403529 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42767
--webserver_interface=127.24.60.254
--webserver_port=34259
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:14.491029 9064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:14.491283 9064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:14.491309 9064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:14.492491 9064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:14.492532 9064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:14.492547 9064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:14.492559 9064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:14.492571 9064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:14.494151 9064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:42767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=34259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:14.494383 9064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:14.494583 9064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:14.504845 9073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.504845 9075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.504871 9072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:14.505213 9064 server_base.cc:1034] running on GCE node
I20260810 07:27:14.505556 9064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:14.505978 9064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:14.507170 9064 hybrid_clock.cc:648] HybridClock initialized: now 1786346834507147 us; error 40 us; skew 500 ppm
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:14.510169 9064 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:14.510651 9064 webserver.cc:460] Webserver started at http://127.24.60.254:34259/ using document root <none> and password file <none>
I20260810 07:27:14.510963 9064 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:14.511034 9064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:14.512661 9064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:14.513684 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.514016 9064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:14.514122 9064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "db9f983567e64c2db9ea68bdd12e5551"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "23c9d91589a2cd7262880ea14a7c92d1"
server_key_iv: "30527c592f7d602acf3774720751f4bc"
server_key_version: "encryptionkey@0"
I20260810 07:27:14.514816 9064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:14.538738 9064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:14.541605 9064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:14.541769 9064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:14.546852 9064 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:42767
I20260810 07:27:14.546893 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:42767 every 8 connection(s)
I20260810 07:27:14.547356 9064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:14.549292 9149 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:14.549998 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9064
I20260810 07:27:14.550212 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap starting.
I20260810 07:27:14.550338 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:40379
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:42767
--webserver_port=37233
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:14.559753 9149 log.cc:824] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:14.562053 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:14.562434 9149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap complete.
I20260810 07:27:14.565049 9149 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:14.565382 9149 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9f983567e64c2db9ea68bdd12e5551, State: Initialized, Role: FOLLOWER
I20260810 07:27:14.565585 9149 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:14.565671 9149 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:14.565733 9149 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:14.565811 9149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:27:14.567008 9149 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:14.567144 9149 leader_election.cc:302] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9f983567e64c2db9ea68bdd12e5551; no voters:
I20260810 07:27:14.567387 9149 leader_election.cc:288] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:27:14.567426 9154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:27:14.567574 9154 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 LEADER]: Becoming Leader. State: Replica: db9f983567e64c2db9ea68bdd12e5551, State: Running, Role: LEADER
I20260810 07:27:14.567688 9149 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:14.567667 9154 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:14.568022 9154 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9f983567e64c2db9ea68bdd12e5551. Latest consensus state: current_term: 2 leader_uuid: "db9f983567e64c2db9ea68bdd12e5551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } } }
I20260810 07:27:14.568107 9154 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:14.568028 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db9f983567e64c2db9ea68bdd12e5551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } } }
I20260810 07:27:14.568375 9162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:14.568373 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:14.569386 9162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:14.569518 9162 catalog_manager.cc:1259] Loaded cluster ID: 5cf2e7917e784b659b26a81c803cbb08
I20260810 07:27:14.569556 9162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:14.569900 9162 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 07:27:14.574563 9162 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 07:27:14.577634 9162 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:14.577894 9162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Loaded TSK: 0
I20260810 07:27:14.578127 9162 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:27:14.639600 9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:14.639945 9152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:14.639998 9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:14.641386 9152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:14.641443 9152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:14.641489 9152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:14.643085 9152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:40379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=37233
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:14.643375 9152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:14.643637 9152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:14.650175 9185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.650175 9184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.650355 9187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:14.650648 9152 server_base.cc:1034] running on GCE node
I20260810 07:27:14.650911 9152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:14.651196 9152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:14.652379 9152 hybrid_clock.cc:648] HybridClock initialized: now 1786346834652364 us; error 33 us; skew 500 ppm
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:14.655287 9152 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:14.655901 9152 webserver.cc:460] Webserver started at http://127.24.60.193:37233/ using document root <none> and password file <none>
I20260810 07:27:14.656155 9152 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:14.656261 9152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:14.657889 9152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:27:14.658627 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.658843 9152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:14.658962 9152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "fb657e7c7e6a4c58b4be58e80416c021"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "749ea85489ba1e5175f51793ed228bd1"
server_key_iv: "70f557b393e29ca6091b6441649b7917"
server_key_version: "encryptionkey@0"
I20260810 07:27:14.659461 9152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:14.673425 9152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:14.676369 9152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:14.676563 9152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:14.676868 9152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:14.677467 9202 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:27:14.679028 9152 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:27:14.679095 9152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 07:27:14.679131 9152 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:27:14.680049 9152 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:27:14.680104 9152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:14.680231 9202 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap starting.
I20260810 07:27:14.687724 9152 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:40379
I20260810 07:27:14.688097 9324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:40379 every 8 connection(s)
I20260810 07:27:14.688233 9152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:14.692121 9202 log.cc:824] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:14.694845 9202 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:14.695217 9202 tablet_bootstrap.cc:492] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap complete.
I20260810 07:27:14.696432 9202 ts_tablet_manager.cc:1397] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260810 07:27:14.697098 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9152
I20260810 07:27:14.697440 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:44699
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:42767
--webserver_port=36549
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:14.698176 9202 raft_consensus.cc:348] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.698472 9202 raft_consensus.cc:729] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb657e7c7e6a4c58b4be58e80416c021, State: Initialized, Role: FOLLOWER
I20260810 07:27:14.698630 9202 consensus_queue.cc:260] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } } peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } }
I20260810 07:27:14.698974 9202 ts_tablet_manager.cc:1428] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:14.705266 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.689063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:41799)
0810 07:27:14.689504 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.689559 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.690411 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.691008 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.691015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.691524 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:14.691833 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.691841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.693656 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.693657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.695084 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.695087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.695964 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.695966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.696125 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.696841 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:14.696851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:14.700173 (+ 3322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:14.702967 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.702979 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.702982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.703369 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.703726 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.703727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.703728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.703894 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.704489 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:14.704497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:14.704666 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.704786 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.704982 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":130,"threads_started":1}
I20260810 07:27:14.706169 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.689247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:41799 (local address 127.24.60.254:42767)
0810 07:27:14.689444 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.689463 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.689711 (+ 248us) server_negotiation.cc:407] Connection header received
0810 07:27:14.690683 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.690690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.690751 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.690917 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:14.691993 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.693529 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.695186 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.695834 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.695975 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.700332 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:14.700338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:14.700338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:14.700355 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:14.702828 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.703490 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.703494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.703494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.703640 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.703992 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.703993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.703994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.704370 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:14.704473 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.704895 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.705068 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260810 07:27:14.707226 9325 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42767
I20260810 07:27:14.707343 9325 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:14.707561 9325 heartbeater.cc:511] Master 127.24.60.254:42767 requested a full tablet report, sending...
I20260810 07:27:14.708478 9102 ts_manager.cc:194] Registered new tserver with Master: fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379)
I20260810 07:27:14.709893 9102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:41799
I20260810 07:27:14.711093 9325 heartbeater.cc:503] Master 127.24.60.254:42767 was elected leader, sending a full tablet report...
I20260810 07:27:14.716094 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.709603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.24.60.193:40379)
0810 07:27:14.710247 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.710251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.711796 (+ 1545us) server_negotiation.cc:407] Connection header received
0810 07:27:14.711935 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.711938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.711954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.712038 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:14.712202 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.713321 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.714909 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.715651 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.715738 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.715809 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.715968 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.716016 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":90,"threads_started":1}
I20260810 07:27:14.716102 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.709292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:40379 (local address 127.0.0.1:53950)
0810 07:27:14.711693 (+ 2401us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.711728 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.711836 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.712056 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.712059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.712061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:14.712131 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.712132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.713423 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.713424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.714809 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.714813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.715755 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.715757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.715844 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.715859 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.715942 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.716003 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2372,"thread_start_us":120,"threads_started":1}
I20260810 07:27:14.716832 9249 tablet_service.cc:1514] Processing DeleteTablet for tablet af5e91a5b6504848b250d2097eecee9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:14 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:53950
I20260810 07:27:14.717572 9335 tablet_replica.cc:330] stopping tablet replica
I20260810 07:27:14.717691 9335 raft_consensus.cc:2229] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:14.717761 9335 raft_consensus.cc:2258] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:14.718176 9335 ts_tablet_manager.cc:1905] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:14.720810 9335 ts_tablet_manager.cc:1918] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:14.720882 9335 log.cc:1196] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/af5e91a5b6504848b250d2097eecee9b
I20260810 07:27:14.721144 9335 ts_tablet_manager.cc:1939] T af5e91a5b6504848b250d2097eecee9b P fb657e7c7e6a4c58b4be58e80416c021: Deleting consensus metadata
I20260810 07:27:14.721901 9086 catalog_manager.cc:4841] TS fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379): tablet af5e91a5b6504848b250d2097eecee9b (table test-table [id=0f0a20868c544d698914a3457b16569b]) successfully deleted
W20260810 07:27:14.791337 9332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:14.791611 9332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:14.791656 9332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:14.792861 9332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:14.792939 9332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:14.792984 9332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:14.794530 9332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:44699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=36549
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:14.794828 9332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:14.795086 9332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:14.800580 9343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.800554 9340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.800545 9341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:14.800911 9332 server_base.cc:1034] running on GCE node
I20260810 07:27:14.801258 9332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:14.801589 9332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:14.802770 9332 hybrid_clock.cc:648] HybridClock initialized: now 1786346834802747 us; error 42 us; skew 500 ppm
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:14.806061 9332 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:14.806628 9332 webserver.cc:460] Webserver started at http://127.24.60.194:36549/ using document root <none> and password file <none>
I20260810 07:27:14.806928 9332 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:14.807008 9332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:14.808867 9332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:27:14.810185 9354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.810482 9332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:14.810612 9332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "66da1aded80947009bcdb49c97346cc3"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "ec7728f348ee570df16f1368fac79246"
server_key_iv: "e8f8bf82d35b3282eafb7edfc7c8f8a4"
server_key_version: "encryptionkey@0"
I20260810 07:27:14.811074 9332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:14.827893 9332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:14.830399 9332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:14.830588 9332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:14.830855 9332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:14.831215 9332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:14.831274 9332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.831321 9332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:14.831344 9332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.837558 9332 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:44699
I20260810 07:27:14.837600 9481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:44699 every 8 connection(s)
I20260810 07:27:14.837980 9332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:14.845422 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9332
I20260810 07:27:14.845715 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:36953
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:42767
--webserver_port=41685
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:14.848469 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.838806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:48217 (local address 127.24.60.254:42767)
0810 07:27:14.838873 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.838875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.839131 (+ 256us) server_negotiation.cc:407] Connection header received
0810 07:27:14.839634 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.839635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.839649 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.839701 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:14.840686 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.841324 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.842397 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.842605 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.842665 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.845129 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:14.845130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:14.845131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:14.845138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:14.846867 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.847417 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.847418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.847419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.847499 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.847757 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.847758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.847758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.847937 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:14.848011 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.848273 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.848383 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:27:14.848688 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.838787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:48217)
0810 07:27:14.838985 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.839026 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.839540 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.839765 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.839769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.840373 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:14.840583 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.840588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.841424 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.841425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.842308 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.842311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.842686 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.842687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.842797 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.843406 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:14.843415 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:14.845027 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:14.846960 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.846970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.846973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.847317 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.847575 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.847575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.847575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.847684 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.848006 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:14.848011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:14.848135 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.848222 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.848381 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260810 07:27:14.849247 9482 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42767
I20260810 07:27:14.849387 9482 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:14.849614 9482 heartbeater.cc:511] Master 127.24.60.254:42767 requested a full tablet report, sending...
I20260810 07:27:14.850067 9102 ts_manager.cc:194] Registered new tserver with Master: 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
I20260810 07:27:14.850466 9102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:48217
W20260810 07:27:14.933691 9485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:14.933969 9485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:14.933997 9485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:14.935250 9485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:14.935297 9485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:14.935325 9485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:14.936859 9485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:36953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=41685
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:14.937141 9485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:14.937433 9485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:14.942765 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:14.942901 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:14.942972 9485 server_base.cc:1034] running on GCE node
W20260810 07:27:14.942760 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:14.943323 9485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:14.943627 9485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:14.944885 9485 hybrid_clock.cc:648] HybridClock initialized: now 1786346834944867 us; error 46 us; skew 500 ppm
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:14.947865 9485 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:14.948437 9485 webserver.cc:460] Webserver started at http://127.24.60.195:41685/ using document root <none> and password file <none>
I20260810 07:27:14.948709 9485 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:14.948776 9485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:14.950565 9485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:14.951445 9503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.951781 9485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:14.951933 9485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "1f03f53f7e4349588b16d484bbb9e436"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "7964412ba16fd1eed016a2c4fcd9034a"
server_key_iv: "9d0f00f63cd205e3f7e7309cd00768f3"
server_key_version: "encryptionkey@0"
I20260810 07:27:14.952487 9485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:14.967959 9485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:14.970644 9485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:14.970804 9485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:14.971104 9485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:14.971464 9485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:14.971519 9485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.971565 9485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:14.971592 9485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:14.978276 9485 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:36953
I20260810 07:27:14.978351 9632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:36953 every 8 connection(s)
I20260810 07:27:14.978746 9485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:14.982439 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9485
Aug 10 07:27:14 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346834, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:14.991140 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.983094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.24.60.254:42767)
0810 07:27:14.983308 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.983310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.983320 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:27:14.983344 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.983344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.983356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.983448 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:14.984270 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.984902 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.986068 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.986275 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.986449 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.987988 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:14.987989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:14.987989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:14.987997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:14.989688 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.990209 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.990210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.990210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.990276 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.990487 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.990487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.990488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.990638 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:14.990810 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.990880 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.990992 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":134,"threads_started":1}
I20260810 07:27:14.991240 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.979895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:59175 (local address 127.24.60.254:42767)
0810 07:27:14.979980 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:14.979982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:14.980243 (+ 261us) server_negotiation.cc:407] Connection header received
0810 07:27:14.980812 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:14.980814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:14.980828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:14.980866 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:14.982015 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.982545 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.984321 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.984535 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.984625 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.987528 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:14.987529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:14.987529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:14.987537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:14.989358 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.989909 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.989911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.989911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.989970 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:14.990343 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:14.990344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:14.990344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:14.990509 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:14.990576 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:14.990945 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 07:27:14.991054 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":50}
I20260810 07:27:14.991707 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:14.979863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:59175)
0810 07:27:14.980102 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:14.980143 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:14.980698 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:14.981013 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:14.981016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:14.981680 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:14.981898 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.981903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.982667 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.982668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.984237 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:14.984241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:14.984639 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:14.984640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:14.984782 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:14.985477 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:14.985489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:14.987402 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:14.989456 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.989472 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.989476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.989830 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.990047 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:14.990048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:14.990048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:14.990229 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:14.990594 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:14.990602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:14.990786 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 07:27:14.990906 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:27:14.991158 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":123,"threads_started":1}
I20260810 07:27:14.992312 9633 heartbeater.cc:347] Connected to a master server at 127.24.60.254:42767
I20260810 07:27:14.992431 9633 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:14.992635 9633 heartbeater.cc:511] Master 127.24.60.254:42767 requested a full tablet report, sending...
I20260810 07:27:14.993044 9102 ts_manager.cc:194] Registered new tserver with Master: 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953)
I20260810 07:27:14.993491 9102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:59175
I20260810 07:27:15.003528 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:15.007670 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.003992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.24.60.254:42767)
0810 07:27:15.004060 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:15.004063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:15.004173 (+ 110us) server_negotiation.cc:407] Connection header received
0810 07:27:15.004242 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:15.004243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:15.004297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:15.004391 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:15.004588 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.005280 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.006471 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.006954 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.007035 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.007312 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:15.007439 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:15.007529 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:15.007567 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 07:27:15.007601 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:27:15.008383 9102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:15.010385 9102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:15.013540 9249 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef6ef6915b24e27abf53144c7da2eae (DEFAULT_TABLE table=test-table [id=dc2479b6423f43ba93eccd5d9c386001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:15.013798 9249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef6ef6915b24e27abf53144c7da2eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:15.015686 9202 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap starting.
I20260810 07:27:15.016058 9202 tablet_bootstrap.cc:654] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:15.016736 9202 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: No bootstrap required, opened a new log
I20260810 07:27:15.016803 9202 ts_tablet_manager.cc:1397] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:15.016920 9202 raft_consensus.cc:348] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.017032 9202 raft_consensus.cc:374] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:15.017102 9202 raft_consensus.cc:729] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb657e7c7e6a4c58b4be58e80416c021, State: Initialized, Role: FOLLOWER
I20260810 07:27:15.017199 9202 consensus_queue.cc:260] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.017319 9202 ts_tablet_manager.cc:1428] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:27:15.017980 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.013286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36953 (local address 127.0.0.1:55268)
0810 07:27:15.013329 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.013340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:15.013389 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:15.014273 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:15.014274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:15.014276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:15.014352 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.014354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.015379 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.015380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.016433 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.016437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.017687 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.017688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.017757 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.017761 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:15.017812 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 07:27:15.017870 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":22}
I20260810 07:27:15.018278 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.013405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.24.60.195:36953)
0810 07:27:15.013977 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:15.013983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:15.013995 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:27:15.014047 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:15.014049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:15.014067 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:15.014345 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:15.014452 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.015267 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.016546 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.017516 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.017583 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.017655 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:15.018138 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 07:27:15.018196 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":145,"threads_started":1}
I20260810 07:27:15.018365 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.013415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44699 (local address 127.0.0.1:49472)
0810 07:27:15.013741 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.013754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:15.013810 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:15.014213 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:15.014215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:15.014218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:15.014347 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.014350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.015728 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.015729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.016894 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.016898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.018105 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.018107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.018173 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.018177 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:15.018240 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:27:15.018287 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
I20260810 07:27:15.018656 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.013685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.24.60.194:44699)
0810 07:27:15.014011 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:15.014015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:15.014024 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:27:15.014064 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:15.014066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:15.014081 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:15.014167 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:15.014453 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.015591 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.017016 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.017994 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.018101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.018177 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:15.018255 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 07:27:15.018297 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":100,"threads_started":1}
I20260810 07:27:15.019090 9559 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef6ef6915b24e27abf53144c7da2eae (DEFAULT_TABLE table=test-table [id=dc2479b6423f43ba93eccd5d9c386001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:15.019467 9559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef6ef6915b24e27abf53144c7da2eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:15.019410 9406 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef6ef6915b24e27abf53144c7da2eae (DEFAULT_TABLE table=test-table [id=dc2479b6423f43ba93eccd5d9c386001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:15.019730 9406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef6ef6915b24e27abf53144c7da2eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:15.022657 9647 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Bootstrap starting.
I20260810 07:27:15.022773 9646 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Bootstrap starting.
I20260810 07:27:15.023456 9647 tablet_bootstrap.cc:654] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:15.023465 9646 tablet_bootstrap.cc:654] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:15.023972 9646 log.cc:824] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:15.024001 9647 log.cc:824] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:15.024725 9647 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: No bootstrap required, opened a new log
I20260810 07:27:15.024741 9646 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: No bootstrap required, opened a new log
I20260810 07:27:15.024783 9647 ts_tablet_manager.cc:1397] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:15.024825 9646 ts_tablet_manager.cc:1397] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:15.026314 9646 raft_consensus.cc:348] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.026365 9647 raft_consensus.cc:348] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.026533 9646 raft_consensus.cc:374] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:15.026582 9646 raft_consensus.cc:729] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f53f7e4349588b16d484bbb9e436, State: Initialized, Role: FOLLOWER
I20260810 07:27:15.026578 9647 raft_consensus.cc:374] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:15.026626 9647 raft_consensus.cc:729] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66da1aded80947009bcdb49c97346cc3, State: Initialized, Role: FOLLOWER
I20260810 07:27:15.026748 9647 consensus_queue.cc:260] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.026741 9646 consensus_queue.cc:260] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.027069 9647 ts_tablet_manager.cc:1428] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 07:27:15.027112 9646 ts_tablet_manager.cc:1428] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 07:27:15.027158 9633 heartbeater.cc:503] Master 127.24.60.254:42767 was elected leader, sending a full tablet report...
I20260810 07:27:15.027215 9482 heartbeater.cc:503] Master 127.24.60.254:42767 was elected leader, sending a full tablet report...
I20260810 07:27:15.050496 9331 raft_consensus.cc:482] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:15.050653 9331 raft_consensus.cc:504] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.051127 9331 leader_election.cc:288] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953), 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
I20260810 07:27:15.055579 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.051514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:56611 (local address 127.24.60.195:36953)
0810 07:27:15.051569 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:15.051571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:15.051598 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:27:15.051638 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:15.051639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:15.051650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:15.051709 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:15.051992 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.052879 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.054509 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.055163 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.055238 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.055246 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:15.055450 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 07:27:15.055504 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 07:27:15.055579 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.051603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53207 (local address 127.24.60.194:44699)
0810 07:27:15.051647 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:15.051649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:15.051820 (+ 171us) server_negotiation.cc:407] Connection header received
0810 07:27:15.051902 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:15.051904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:15.051918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:15.051982 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:15.052145 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.052936 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.054509 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.055120 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.055216 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.055229 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:15.055450 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:27:15.055504 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:27:15.055581 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.051382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36953 (local address 127.24.60.193:56611)
0810 07:27:15.051453 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.051469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:15.051542 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:15.051722 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:15.051723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:15.051725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:15.051832 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.051834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.052965 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.052965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.054382 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.054386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.055244 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.055245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.055338 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.055341 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:27:15.055400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 07:27:15.055451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":90}
I20260810 07:27:15.055593 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.051524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44699 (local address 127.24.60.193:53207)
0810 07:27:15.051742 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.051753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:15.051817 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:15.052003 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:15.052003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:15.052005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:15.052067 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.052068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.053014 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.053014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.054386 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:15.054387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.055257 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.055258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:15.055334 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.055338 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:15.055426 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 07:27:15.055457 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":33,"thread_start_us":153,"threads_started":1}
I20260810 07:27:15.055980 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef6ef6915b24e27abf53144c7da2eae" candidate_uuid: "fb657e7c7e6a4c58b4be58e80416c021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f53f7e4349588b16d484bbb9e436" is_pre_election: true
I20260810 07:27:15.056205 9580 raft_consensus.cc:2454] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb657e7c7e6a4c58b4be58e80416c021 in term 0.
I20260810 07:27:15.056331 9427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef6ef6915b24e27abf53144c7da2eae" candidate_uuid: "fb657e7c7e6a4c58b4be58e80416c021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66da1aded80947009bcdb49c97346cc3" is_pre_election: true
I20260810 07:27:15.056496 9427 raft_consensus.cc:2454] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb657e7c7e6a4c58b4be58e80416c021 in term 0.
I20260810 07:27:15.056504 9196 leader_election.cc:302] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f53f7e4349588b16d484bbb9e436, fb657e7c7e6a4c58b4be58e80416c021; no voters:
I20260810 07:27:15.056682 9331 raft_consensus.cc:2781] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:15.056770 9331 raft_consensus.cc:482] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:15.056802 9331 raft_consensus.cc:3037] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:15.057696 9331 raft_consensus.cc:504] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.057844 9331 leader_election.cc:288] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 1 election: Requested vote from peers 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953), 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
I20260810 07:27:15.058040 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef6ef6915b24e27abf53144c7da2eae" candidate_uuid: "fb657e7c7e6a4c58b4be58e80416c021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f53f7e4349588b16d484bbb9e436"
I20260810 07:27:15.058085 9427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef6ef6915b24e27abf53144c7da2eae" candidate_uuid: "fb657e7c7e6a4c58b4be58e80416c021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66da1aded80947009bcdb49c97346cc3"
I20260810 07:27:15.058123 9580 raft_consensus.cc:3037] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:15.058189 9427 raft_consensus.cc:3037] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:15.058917 9580 raft_consensus.cc:2454] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb657e7c7e6a4c58b4be58e80416c021 in term 1.
I20260810 07:27:15.059101 9196 leader_election.cc:302] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f53f7e4349588b16d484bbb9e436, fb657e7c7e6a4c58b4be58e80416c021; no voters:
I20260810 07:27:15.059101 9427 raft_consensus.cc:2454] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb657e7c7e6a4c58b4be58e80416c021 in term 1.
I20260810 07:27:15.059213 9331 raft_consensus.cc:2781] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:15.059427 9331 raft_consensus.cc:686] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 LEADER]: Becoming Leader. State: Replica: fb657e7c7e6a4c58b4be58e80416c021, State: Running, Role: LEADER
I20260810 07:27:15.059525 9331 consensus_queue.cc:237] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.060389 9102 catalog_manager.cc:5495] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "fb657e7c7e6a4c58b4be58e80416c021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } health_report { overall_health: HEALTHY } } }
I20260810 07:27:15.078660 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.075913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.24.60.193:40379)
0810 07:27:15.075994 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:15.075996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:15.076007 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:15.076053 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:15.076054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:15.076076 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:15.076178 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:15.076375 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.076932 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.077831 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:15.078054 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:15.078150 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:15.078345 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:15.078465 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:15.078532 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:15.078570 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 07:27:15.078598 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:27:15.080555 9580 raft_consensus.cc:1261] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Refusing update from remote peer fb657e7c7e6a4c58b4be58e80416c021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:15.080556 9427 raft_consensus.cc:1261] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Refusing update from remote peer fb657e7c7e6a4c58b4be58e80416c021: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:15.081215 9331 consensus_queue.cc:1035] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:15.081576 9653 consensus_queue.cc:1035] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:15.084263 9656 mvcc.cc:205] Tried to move back new op lower bound from 7316876636486590464 to 7316876636404588544. Current Snapshot: MvccSnapshot[applied={T|T < 7316876636486590464}]
I20260810 07:27:15.084753 9659 mvcc.cc:205] Tried to move back new op lower bound from 7316876636486590464 to 7316876636404588544. Current Snapshot: MvccSnapshot[applied={T|T < 7316876636486590464}]
I20260810 07:27:15.087348 9100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:15.087479 9100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:15.088327 9100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Sending DeleteTablet for 3 replicas of tablet 0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:15.088730 9249 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef6ef6915b24e27abf53144c7da2eae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:15 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:53950
I20260810 07:27:15.088840 9335 tablet_replica.cc:330] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: stopping tablet replica
I20260810 07:27:15.088897 9335 raft_consensus.cc:2229] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:15.088977 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 9152
I20260810 07:27:15.088995 9335 raft_consensus.cc:2258] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:15.088881 9406 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef6ef6915b24e27abf53144c7da2eae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:15 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:49472
I20260810 07:27:15.089135 9559 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef6ef6915b24e27abf53144c7da2eae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:15 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:55268
I20260810 07:27:15.092752 9668 tablet_replica.cc:330] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: stopping tablet replica
I20260810 07:27:15.092988 9668 raft_consensus.cc:2229] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:15.093214 9668 raft_consensus.cc:2258] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:15.094264 9668 ts_tablet_manager.cc:1905] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:15.095623 9666 tablet_replica.cc:330] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: stopping tablet replica
I20260810 07:27:15.095783 9666 raft_consensus.cc:2229] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:15.095929 9666 raft_consensus.cc:2258] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:15.096487 9666 ts_tablet_manager.cc:1905] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:15.096686 9668 ts_tablet_manager.cc:1918] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:15.096791 9668 log.cc:1196] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:15.097198 9668 ts_tablet_manager.cc:1939] T 0ef6ef6915b24e27abf53144c7da2eae P 1f03f53f7e4349588b16d484bbb9e436: Deleting consensus metadata
I20260810 07:27:15.097792 9085 catalog_manager.cc:4841] TS 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953): tablet 0ef6ef6915b24e27abf53144c7da2eae (table test-table [id=dc2479b6423f43ba93eccd5d9c386001]) successfully deleted
I20260810 07:27:15.098713 9666 ts_tablet_manager.cc:1918] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:15.098801 9666 log.cc:1196] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:15.099180 9666 ts_tablet_manager.cc:1939] T 0ef6ef6915b24e27abf53144c7da2eae P 66da1aded80947009bcdb49c97346cc3: Deleting consensus metadata
I20260810 07:27:15.099725 9085 catalog_manager.cc:4841] TS 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699): tablet 0ef6ef6915b24e27abf53144c7da2eae (table test-table [id=dc2479b6423f43ba93eccd5d9c386001]) successfully deleted
W20260810 07:27:15.101428 9086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:40379 (error 108)
I20260810 07:27:15.101708 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 9332
I20260810 07:27:15.102043 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.101903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:40379 (local address 127.0.0.1:53970)
0810 07:27:15.101961 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.102001 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:40379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 07:27:15.102162 9086 catalog_manager.cc:4584] TS fb657e7c7e6a4c58b4be58e80416c021 (127.24.60.193:40379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ef6ef6915b24e27abf53144c7da2eae: Network error: Client connection negotiation failed: client connection to 127.24.60.193:40379: connect: Connection refused (error 111)
I20260810 07:27:15.113229 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 9485
I20260810 07:27:15.121726 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 9064
I20260810 07:27:15.131338 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42767
--webserver_interface=127.24.60.254
--webserver_port=34259
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:15.221933 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:15.222226 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:15.222270 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:15.223830 9670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:15.223905 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:15.223927 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:15.223948 9670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:15.223970 9670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:15.225847 9670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:42767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=34259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:15.226195 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:15.226459 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:15.231848 9680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.231853 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.231848 9682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:15.232448 9670 server_base.cc:1034] running on GCE node
I20260810 07:27:15.232736 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:15.233093 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:15.234316 9670 hybrid_clock.cc:648] HybridClock initialized: now 1786346835234292 us; error 39 us; skew 500 ppm
Aug 10 07:27:15 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346835, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:15.237478 9670 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:15.238060 9670 webserver.cc:460] Webserver started at http://127.24.60.254:34259/ using document root <none> and password file <none>
I20260810 07:27:15.238343 9670 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:15.238399 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:15.240113 9670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:27:15.241070 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.241329 9670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:15.241480 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "db9f983567e64c2db9ea68bdd12e5551"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "23c9d91589a2cd7262880ea14a7c92d1"
server_key_iv: "30527c592f7d602acf3774720751f4bc"
server_key_version: "encryptionkey@0"
I20260810 07:27:15.241928 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:15.257261 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:15.259632 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:15.259836 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:15.264544 9670 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:42767
I20260810 07:27:15.264608 9750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:42767 every 8 connection(s)
I20260810 07:27:15.265053 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:15.266834 9752 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:15.267488 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap starting.
I20260810 07:27:15.267781 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9670
I20260810 07:27:15.268128 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:40379
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:42767
--webserver_port=37233
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:15.277614 9752 log.cc:824] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:15.280066 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:15.280452 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551: Bootstrap complete.
I20260810 07:27:15.283115 9752 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:15.283375 9752 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9f983567e64c2db9ea68bdd12e5551, State: Initialized, Role: FOLLOWER
I20260810 07:27:15.283558 9752 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:15.283635 9752 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:15.283701 9752 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:15.283747 9752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:27:15.284857 9752 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:15.284992 9752 leader_election.cc:302] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9f983567e64c2db9ea68bdd12e5551; no voters:
I20260810 07:27:15.285233 9752 leader_election.cc:288] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 07:27:15.285307 9756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 07:27:15.285482 9756 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [term 3 LEADER]: Becoming Leader. State: Replica: db9f983567e64c2db9ea68bdd12e5551, State: Running, Role: LEADER
I20260810 07:27:15.285573 9752 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:15.285588 9756 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } }
I20260810 07:27:15.285894 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9f983567e64c2db9ea68bdd12e5551. Latest consensus state: current_term: 3 leader_uuid: "db9f983567e64c2db9ea68bdd12e5551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } } }
I20260810 07:27:15.285982 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:15.286187 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "db9f983567e64c2db9ea68bdd12e5551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9f983567e64c2db9ea68bdd12e5551" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 42767 } } }
I20260810 07:27:15.286247 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9f983567e64c2db9ea68bdd12e5551 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:15.286768 9760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:15.287823 9760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:15.287948 9760 catalog_manager.cc:1259] Loaded cluster ID: 5cf2e7917e784b659b26a81c803cbb08
I20260810 07:27:15.287995 9760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 07:27:15.288424 9760 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbc265519ad google::LogMessage::Fail() at ??:0
@ 0x7fbc2655488a google::LogMessage::SendToLog() at ??:0
@ 0x7fbc265514de google::LogMessage::Flush() at ??:0
@ 0x7fbc26552a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbc2e4720f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbc2e47b8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbc2841f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbc28414954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbc292786db start_thread at ??:0
@ 0x7fbc2504f61f clone at ??:0
W20260810 07:27:15.371467 9754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:15.371766 9754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:15.371815 9754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:15.373301 9754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:15.373379 9754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:15.373430 9754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:15.375237 9754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:40379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=37233
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:15.375564 9754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:15.375898 9754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:15.381916 9798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.381947 9796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.381966 9795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:15.383737 9754 server_base.cc:1034] running on GCE node
I20260810 07:27:15.384162 9754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:15.384577 9754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:15.385712 9754 hybrid_clock.cc:648] HybridClock initialized: now 1786346835385682 us; error 49 us; skew 500 ppm
Aug 10 07:27:15 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346835, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:15.392176 9754 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:15.392892 9754 webserver.cc:460] Webserver started at http://127.24.60.193:37233/ using document root <none> and password file <none>
I20260810 07:27:15.393239 9754 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:15.393316 9754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:15.395061 9754 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:27:15.395905 9806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.396163 9754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:15.396276 9754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "fb657e7c7e6a4c58b4be58e80416c021"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "749ea85489ba1e5175f51793ed228bd1"
server_key_iv: "70f557b393e29ca6091b6441649b7917"
server_key_version: "encryptionkey@0"
I20260810 07:27:15.396673 9754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:15.404702 9754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:15.407732 9754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:15.407876 9754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:15.408172 9754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:15.408743 9813 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:27:15.411142 9754 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:27:15.411300 9754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260810 07:27:15.411396 9754 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:27:15.412451 9754 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:27:15.412534 9754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:15.412585 9813 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap starting.
I20260810 07:27:15.420661 9754 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:40379
I20260810 07:27:15.421109 9754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:15.421514 9936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:40379 every 8 connection(s)
I20260810 07:27:15.424798 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9754
I20260810 07:27:15.425014 9813 log.cc:824] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:15.425235 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:44699
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:42767
--webserver_port=36549
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:15.428186 9813 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:15.428560 9813 tablet_bootstrap.cc:492] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Bootstrap complete.
I20260810 07:27:15.430012 9813 ts_tablet_manager.cc:1397] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260810 07:27:15.431797 9813 raft_consensus.cc:348] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.432233 9813 raft_consensus.cc:729] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb657e7c7e6a4c58b4be58e80416c021, State: Initialized, Role: FOLLOWER
I20260810 07:27:15.432461 9813 consensus_queue.cc:260] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:15.432811 9813 ts_tablet_manager.cc:1428] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260810 07:27:15.509222 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:27:15.421892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:59437)
0810 07:27:15.422101 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.422149 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:15.422692 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:15.508950 (+ 86258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":126,"threads_started":1}
W20260810 07:27:15.509564 9937 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:42767 (0 consecutive failures): Network error: Failed to ping master at 127.24.60.254:42767: Client connection negotiation failed: client connection to 127.24.60.254:42767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:27:15.509999 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.509838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:48289)
0810 07:27:15.509904 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.509964 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:27:15.510417 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.510298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:43147)
0810 07:27:15.510354 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.510390 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 07:27:15.510572 9937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:15.519673 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:15.519966 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:15.519999 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:15.521469 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:15.521544 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:15.521582 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:15.523485 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:44699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=36549
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:15.523847 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:15.524150 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:15.530303 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.530303 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.530308 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:15.530441 9941 server_base.cc:1034] running on GCE node
I20260810 07:27:15.530800 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:15.531093 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:15.532276 9941 hybrid_clock.cc:648] HybridClock initialized: now 1786346835532245 us; error 45 us; skew 500 ppm
Aug 10 07:27:15 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346835, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:15.535445 9941 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:15.536010 9941 webserver.cc:460] Webserver started at http://127.24.60.194:36549/ using document root <none> and password file <none>
I20260810 07:27:15.536248 9941 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:15.536299 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:15.537973 9941 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:27:15.538622 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.538829 9941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:15.538932 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "66da1aded80947009bcdb49c97346cc3"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "ec7728f348ee570df16f1368fac79246"
server_key_iv: "e8f8bf82d35b3282eafb7edfc7c8f8a4"
server_key_version: "encryptionkey@0"
I20260810 07:27:15.539319 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:15.553215 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:15.555728 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:15.555896 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:15.556145 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:15.556468 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:15.556504 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.556530 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:15.556545 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.563594 9941 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:44699
I20260810 07:27:15.563684 10095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:44699 every 8 connection(s)
I20260810 07:27:15.564083 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:15.565412 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.564916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:55163)
0810 07:27:15.565107 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.565252 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
W20260810 07:27:15.565903 10096 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:42767 (0 consecutive failures): Network error: Failed to ping master at 127.24.60.254:42767: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
I20260810 07:27:15.566272 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.566170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:41573)
0810 07:27:15.566222 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.566252 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 07:27:15.566749 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.566622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:45599)
0810 07:27:15.566695 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.566726 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 07:27:15.567108 10096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:27:15.572016 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 9941
I20260810 07:27:15.572378 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:36953
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:42767
--webserver_port=41685
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:15.660832 10100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:15.661083 10100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:15.661113 10100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:15.662426 10100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:15.662489 10100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:15.662524 10100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:15.664161 10100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:36953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=41685
--tserver_master_addrs=127.24.60.254:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:15.664495 10100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:15.664779 10100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:15.670667 10110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:15.670627 10113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:15.670863 10100 server_base.cc:1034] running on GCE node
W20260810 07:27:15.670655 10111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:15.671263 10100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:15.671557 10100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:15.672739 10100 hybrid_clock.cc:648] HybridClock initialized: now 1786346835672730 us; error 47 us; skew 500 ppm
Aug 10 07:27:15 dist-test-slave-vxj2 krb5kdc[8401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346835, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:15.675731 10100 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:15.676267 10100 webserver.cc:460] Webserver started at http://127.24.60.195:41685/ using document root <none> and password file <none>
I20260810 07:27:15.676553 10100 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:15.676630 10100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:15.678161 10100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:15.678987 10121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.679248 10100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:15.679368 10100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "1f03f53f7e4349588b16d484bbb9e436"
format_stamp: "Formatted at 2026-08-10 07:27:13 on dist-test-slave-vxj2"
server_key: "7964412ba16fd1eed016a2c4fcd9034a"
server_key_iv: "9d0f00f63cd205e3f7e7309cd00768f3"
server_key_version: "encryptionkey@0"
I20260810 07:27:15.679764 10100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:15.692023 10100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:15.694957 10100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:15.695166 10100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:15.695483 10100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:15.695879 10100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:15.695942 10100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.695995 10100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:15.696029 10100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:15.703897 10100 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:36953
I20260810 07:27:15.704056 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:36953 every 8 connection(s)
I20260810 07:27:15.704456 10100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:15.706233 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.705449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:40369)
0810 07:27:15.705657 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.705760 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
W20260810 07:27:15.706627 10251 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:42767 (0 consecutive failures): Network error: Failed to ping master at 127.24.60.254:42767: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
I20260810 07:27:15.706987 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.706873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:49503)
0810 07:27:15.706928 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.706960 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 07:27:15.707307 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:15.707227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:49915)
0810 07:27:15.707267 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:15.707292 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 07:27:15.707438 10251 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:27:15.709002 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 10100
I20260810 07:27:16.511471 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:16.511090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:34803)
0810 07:27:16.511339 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:16.511403 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260810 07:27:16.567885 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:16.567591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:60405)
0810 07:27:16.567753 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:16.567816 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260810 07:27:16.708420 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:16.707978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:41323)
0810 07:27:16.708218 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:16.708336 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":143,"threads_started":1}
I20260810 07:27:17.111459 10260 raft_consensus.cc:482] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:17.111616 10260 raft_consensus.cc:504] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:17.112154 10260 leader_election.cc:288] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953), 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
Aug 10 07:27:17 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346835, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.194@KRBTEST.COM
Aug 10 07:27:17 dist-test-slave-vxj2 krb5kdc[8401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346835, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.195@KRBTEST.COM
I20260810 07:27:17.129077 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.112588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:36235 (local address 127.24.60.195:36953)
0810 07:27:17.112782 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:17.112798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:17.112840 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:27:17.113612 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:17.113615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:17.113667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:17.113849 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:17.114751 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.115881 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.117007 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.117496 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.117611 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:17.124870 (+ 7259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:17.124876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:17.124877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:17.124895 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:17.127130 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:17.127636 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:17.127640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:17.127641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:17.127776 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:17.128090 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:17.128091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:17.128091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:17.128384 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:17.128495 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:17.128811 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 07:27:17.128953 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260810 07:27:17.129089 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.112899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:34603 (local address 127.24.60.194:44699)
0810 07:27:17.113091 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:17.113103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:17.113124 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:27:17.113638 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:17.113641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:17.113681 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:17.113814 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:17.114764 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.115983 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.116970 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.117511 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.117657 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:17.124574 (+ 6917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:17.124580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:17.124581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:17.124597 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:17.126964 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:17.127636 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:17.127640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:17.127641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:17.127788 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:17.128084 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:17.128090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:17.128090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:17.128397 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:17.128498 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:17.128807 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 07:27:17.128946 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":122,"threads_started":1}
I20260810 07:27:17.129065 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.112426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36953 (local address 127.24.60.193:36235)
0810 07:27:17.112757 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:17.112770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:17.112831 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:17.113933 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:17.113934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:17.114379 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:17.114594 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:17.114601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.115991 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.115993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:17.116853 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:17.116856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.117598 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.117600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:17.117731 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:17.122456 (+ 4725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:17.122462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:17.124717 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:17.127218 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:17.127219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:17.127219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:17.127530 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:17.127898 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:17.127899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:17.127899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:17.128001 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:17.128487 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:17.128495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:17.128630 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 07:27:17.128737 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 07:27:17.128967 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":80,"thread_start_us":104,"threads_started":1}
I20260810 07:27:17.129228 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.112657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44699 (local address 127.24.60.193:34603)
0810 07:27:17.112772 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:17.112786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:17.112873 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:17.113926 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:17.113929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:17.114413 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:17.114594 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:17.114601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.116093 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.116094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:17.116850 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:17.116852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:17.117626 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:17.117627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:17.117729 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:17.122413 (+ 4684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:17.122426 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:17.124354 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:17.127069 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:17.127080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:17.127082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:17.127530 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:17.127874 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:17.127874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:17.127875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:17.128000 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:17.128492 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:17.128500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:17.128629 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 07:27:17.128739 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 07:27:17.129058 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":135,"thread_start_us":42,"threads_started":1}
W20260810 07:27:17.130116 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
W20260810 07:27:17.130685 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:17.130785 9807 leader_election.cc:302] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb657e7c7e6a4c58b4be58e80416c021; no voters: 1f03f53f7e4349588b16d484bbb9e436, 66da1aded80947009bcdb49c97346cc3
I20260810 07:27:17.130977 10260 raft_consensus.cc:2726] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:17.512313 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.512122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:34789)
0810 07:27:17.512217 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:17.512266 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 07:27:17.568924 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.568599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:59715)
0810 07:27:17.568795 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:17.568861 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260810 07:27:17.709314 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:17.708956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:44551)
0810 07:27:17.709126 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:17.709250 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260810 07:27:18.513439 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:18.513087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:54611)
0810 07:27:18.513313 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:18.513378 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":148,"threads_started":1}
I20260810 07:27:18.569859 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:18.569578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:57425)
0810 07:27:18.569740 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:18.569804 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260810 07:27:18.710325 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:18.709989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:41307)
0810 07:27:18.710204 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:18.710265 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20260810 07:27:19.259109 10271 raft_consensus.cc:482] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:19.259258 10271 raft_consensus.cc:504] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:19.259582 10271 leader_election.cc:288] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953), 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
W20260810 07:27:19.260181 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
W20260810 07:27:19.260275 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:19.260295 9807 leader_election.cc:302] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb657e7c7e6a4c58b4be58e80416c021; no voters: 1f03f53f7e4349588b16d484bbb9e436, 66da1aded80947009bcdb49c97346cc3
I20260810 07:27:19.260368 10271 raft_consensus.cc:2726] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:19.514514 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:19.514173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:40339)
0810 07:27:19.514388 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:19.514443 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260810 07:27:19.570840 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:19.570561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:34667)
0810 07:27:19.570722 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:19.570766 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260810 07:27:19.711199 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:19.710926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:49783)
0810 07:27:19.711094 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:19.711138 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260810 07:27:20.515390 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:20.515168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:35451)
0810 07:27:20.515300 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:20.515341 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260810 07:27:20.571766 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:20.571453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:37379)
0810 07:27:20.571668 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:20.571715 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260810 07:27:20.712260 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:20.711961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:37681)
0810 07:27:20.712154 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:20.712205 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260810 07:27:21.516342 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:21.516072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:42695)
0810 07:27:21.516238 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:21.516289 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260810 07:27:21.572818 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:21.572514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:53561)
0810 07:27:21.572699 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:21.572763 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260810 07:27:21.713344 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:21.712952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:33171)
0810 07:27:21.713141 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:21.713271 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260810 07:27:22.134384 10282 raft_consensus.cc:482] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:22.134485 10282 raft_consensus.cc:504] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:22.134662 10282 leader_election.cc:288] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953), 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
W20260810 07:27:22.135277 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
W20260810 07:27:22.135375 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:22.135406 9807 leader_election.cc:302] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb657e7c7e6a4c58b4be58e80416c021; no voters: 1f03f53f7e4349588b16d484bbb9e436, 66da1aded80947009bcdb49c97346cc3
I20260810 07:27:22.135535 10282 raft_consensus.cc:2726] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:22.517261 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:22.516963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:52585)
0810 07:27:22.517110 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:22.517204 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260810 07:27:22.573803 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:22.573473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:35243)
0810 07:27:22.573654 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:22.573718 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260810 07:27:22.714246 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:22.713932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:53637)
0810 07:27:22.714145 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:22.714196 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260810 07:27:23.518148 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:23.517875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:46887)
0810 07:27:23.518042 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:23.518094 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260810 07:27:23.574899 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:23.574634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:34505)
0810 07:27:23.574804 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:23.574847 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260810 07:27:23.715206 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:23.714919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:56227)
0810 07:27:23.715102 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:23.715155 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260810 07:27:24.519165 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:24.518845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:44471)
0810 07:27:24.519064 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:24.519117 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260810 07:27:24.575865 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:24.575503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:53719)
0810 07:27:24.575690 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:24.575745 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260810 07:27:24.637081 10292 raft_consensus.cc:482] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:24.637235 10292 raft_consensus.cc:504] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f03f53f7e4349588b16d484bbb9e436" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36953 } } peers { permanent_uuid: "66da1aded80947009bcdb49c97346cc3" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44699 } } peers { permanent_uuid: "fb657e7c7e6a4c58b4be58e80416c021" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 40379 } }
I20260810 07:27:24.637409 10292 leader_election.cc:288] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953), 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699)
W20260810 07:27:24.637822 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f03f53f7e4349588b16d484bbb9e436 (127.24.60.195:36953): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
W20260810 07:27:24.637905 9807 leader_election.cc:341] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66da1aded80947009bcdb49c97346cc3 (127.24.60.194:44699): Not found: Tablet not found: 0ef6ef6915b24e27abf53144c7da2eae
I20260810 07:27:24.637943 9807 leader_election.cc:302] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb657e7c7e6a4c58b4be58e80416c021; no voters: 1f03f53f7e4349588b16d484bbb9e436, 66da1aded80947009bcdb49c97346cc3
I20260810 07:27:24.638015 10292 raft_consensus.cc:2726] T 0ef6ef6915b24e27abf53144c7da2eae P fb657e7c7e6a4c58b4be58e80416c021 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:24.716279 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:24.715977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:41193)
0810 07:27:24.716171 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:24.716227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260810 07:27:25.520205 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:25.519813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:40085)
0810 07:27:25.520039 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:25.520146 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260810 07:27:25.576798 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:25.576540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:32863)
0810 07:27:25.576689 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:25.576746 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260810 07:27:25.717350 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:25.716937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:36697)
0810 07:27:25.717234 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:25.717284 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260810 07:27:26.521059 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:26.520809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:58725)
0810 07:27:26.520973 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:26.521010 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260810 07:27:26.577735 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:26.577474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:53513)
0810 07:27:26.577634 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:26.577684 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260810 07:27:26.718317 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:26.718010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:37071)
0810 07:27:26.718210 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:26.718263 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260810 07:27:27.521966 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:27.521661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:39471)
0810 07:27:27.521824 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:27.521871 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260810 07:27:27.578675 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:27.578401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:46279)
0810 07:27:27.578572 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:27.578625 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260810 07:27:27.719321 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:27.719039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:38273)
0810 07:27:27.719231 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:27.719272 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":123,"threads_started":1}
I20260810 07:27:28.522895 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:28.522598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:41239)
0810 07:27:28.522751 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:28.522844 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260810 07:27:28.579643 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:28.579353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:44975)
0810 07:27:28.579541 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:28.579591 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260810 07:27:28.720320 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:28.720013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:52301)
0810 07:27:28.720207 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:28.720258 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260810 07:27:29.523746 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:29.523498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:57745)
0810 07:27:29.523647 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:29.523696 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260810 07:27:29.580518 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:29.580182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:51031)
0810 07:27:29.580417 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:29.580463 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260810 07:27:29.721405 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:29.721047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:35609)
0810 07:27:29.721309 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:29.721357 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":162,"threads_started":1}
I20260810 07:27:30.524789 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:30.524525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.193:44987)
0810 07:27:30.524696 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:30.524740 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260810 07:27:30.581724 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:30.581408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.194:45605)
0810 07:27:30.581619 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:30.581671 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20260810 07:27:30.715240 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 9754
I20260810 07:27:30.722362 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:30.722113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:42767 (local address 127.24.60.195:51601)
0810 07:27:30.722272 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:30.722312 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260810 07:27:30.723940 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 9941
I20260810 07:27:30.732942 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 10100
2026-08-10T07:27:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18680 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:30 dist-test-slave-vxj2 krb5kdc[10318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:30 dist-test-slave-vxj2 krb5kdc[10318](info): set up 2 sockets
Aug 10 07:27:30 dist-test-slave-vxj2 krb5kdc[10318](info): commencing operation
krb5kdc: starting...
W20260810 07:27:31.917114 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:27:32.053423 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36423
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:32.141515 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:32.141786 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:32.141813 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:32.143193 10332 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 07:27:32.143267 10332 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 07:27:32.143285 10332 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 07:27:32.143297 10332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:32.143311 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:32.143325 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:32.143338 10332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:32.143350 10332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:32.145093 10332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:32.145457 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:32.145704 10332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:32.151997 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.151916 10339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:32.151950 10332 server_base.cc:1034] running on GCE node
W20260810 07:27:32.151903 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:32.152654 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:32.153050 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:32.154238 10332 hybrid_clock.cc:648] HybridClock initialized: now 1786346852154236 us; error 45 us; skew 500 ppm
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:32.157433 10332 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:32.157996 10332 webserver.cc:460] Webserver started at http://127.24.60.254:33487/ using document root <none> and password file <none>
I20260810 07:27:32.158281 10332 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:32.158335 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:32.158509 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:32.159812 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "d8bdb127b83545bba13d22e4f3167b4c"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "ed065666ac2a024edbe32d5094e583de"
server_key_iv: "ef27cdf664a48c11361923bf87af4159"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.160271 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "d8bdb127b83545bba13d22e4f3167b4c"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "ed065666ac2a024edbe32d5094e583de"
server_key_iv: "ef27cdf664a48c11361923bf87af4159"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.162063 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 07:27:32.163126 10350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.163437 10332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:32.163565 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "d8bdb127b83545bba13d22e4f3167b4c"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "ed065666ac2a024edbe32d5094e583de"
server_key_iv: "ef27cdf664a48c11361923bf87af4159"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.163654 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:32.185026 10332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:32.187763 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:32.188060 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:32.192750 10332 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:36423
I20260810 07:27:32.192821 10415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:36423 every 8 connection(s)
I20260810 07:27:32.193264 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:32.194417 10416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:32.197908 10416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap starting.
I20260810 07:27:32.198586 10416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:32.198890 10416 log.cc:824] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:32.199971 10416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: No bootstrap required, opened a new log
I20260810 07:27:32.200022 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 10332
I20260810 07:27:32.200117 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:27:32.200347 24819 external_mini_cluster.cc:1428] Setting key c72c7c4c86002864f1c9077abecfa9f4
I20260810 07:27:32.202672 10416 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:32.202860 10416 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:32.202885 10416 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8bdb127b83545bba13d22e4f3167b4c, State: Initialized, Role: FOLLOWER
I20260810 07:27:32.203059 10416 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:32.203166 10416 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:32.203207 10416 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:32.203255 10416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:32.203977 10416 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:32.204128 10416 leader_election.cc:302] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8bdb127b83545bba13d22e4f3167b4c; no voters:
I20260810 07:27:32.204355 10416 leader_election.cc:288] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:32.204391 10423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:32.204524 10423 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 LEADER]: Becoming Leader. State: Replica: d8bdb127b83545bba13d22e4f3167b4c, State: Running, Role: LEADER
I20260810 07:27:32.204681 10416 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:32.204694 10423 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:32.205406 10425 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bdb127b83545bba13d22e4f3167b4c. Latest consensus state: current_term: 1 leader_uuid: "d8bdb127b83545bba13d22e4f3167b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } } }
I20260810 07:27:32.205600 10425 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:32.205433 10424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8bdb127b83545bba13d22e4f3167b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } } }
I20260810 07:27:32.205655 10424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:32.205855 10434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:32.206427 10434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:32.208897 10434 catalog_manager.cc:1347] Generated new cluster ID: 48e45ffb1f19431d8dd4eea1a107510b
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:32.208964 10434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:32.213850 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.200957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.24.60.254:36423)
0810 07:27:32.201192 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.201210 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.201241 (+ 31us) server_negotiation.cc:407] Connection header received
0810 07:27:32.201696 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.201704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.201761 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.201931 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:32.202677 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.204166 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.205335 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.206097 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.206240 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.209558 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:32.209564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:32.209564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:32.209579 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:32.211743 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.212197 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.212201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.212201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.212332 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.212552 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.212553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.212553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.212916 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:32.213092 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.213305 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.213468 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260810 07:27:32.242343 10434 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:32.243014 10434 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:32.256245 10434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Generated new TSK 0
I20260810 07:27:32.256549 10434 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:27:32.289137 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36423
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:32.399113 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:32.399374 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:32.399415 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:32.400802 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:32.401013 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:32.401073 10447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:32.403474 10447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:32.403843 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:32.404193 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:32.411008 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.411027 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.411001 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:32.411001 10447 server_base.cc:1034] running on GCE node
I20260810 07:27:32.411487 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:32.411809 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:32.413023 10447 hybrid_clock.cc:648] HybridClock initialized: now 1786346852412982 us; error 122 us; skew 500 ppm
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:32.419795 10447 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:32.420424 10447 webserver.cc:460] Webserver started at http://127.24.60.193:40525/ using document root <none> and password file <none>
I20260810 07:27:32.420691 10447 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:32.420768 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:32.420948 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:32.422188 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "2106bd07c93d472781a063404f6a0586"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "badf5ce58093fe8680e3c46c18cea086"
server_key_iv: "86dd70271f866013d5c0179b7e3ed140"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.422603 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "2106bd07c93d472781a063404f6a0586"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "badf5ce58093fe8680e3c46c18cea086"
server_key_iv: "86dd70271f866013d5c0179b7e3ed140"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.424134 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:27:32.425055 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.425287 10447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:32.425442 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "2106bd07c93d472781a063404f6a0586"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "badf5ce58093fe8680e3c46c18cea086"
server_key_iv: "86dd70271f866013d5c0179b7e3ed140"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.425530 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:32.446746 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:32.449368 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:32.449590 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:32.449882 10447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:32.450244 10447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:32.450310 10447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.450356 10447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:32.450385 10447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.457037 10447 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46173
I20260810 07:27:32.457085 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46173 every 8 connection(s)
I20260810 07:27:32.457549 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:32.467195 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 10447
I20260810 07:27:32.467399 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:27:32.467650 24819 external_mini_cluster.cc:1428] Setting key 90f576cfaab9d4acaac9ee4632e48aac
I20260810 07:27:32.469554 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.458590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:41303 (local address 127.24.60.254:36423)
0810 07:27:32.458709 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.458712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.458932 (+ 220us) server_negotiation.cc:407] Connection header received
0810 07:27:32.459576 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.459577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.459591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.459657 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:32.460799 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.461430 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.462508 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.462727 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.462829 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.465381 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:32.465383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:32.465383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:32.465394 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:32.467324 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.467908 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.467909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.467910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.468011 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.468623 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.468624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.468625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.468795 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:32.468880 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.469312 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.469458 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 07:27:32.469740 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.458486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:41303)
0810 07:27:32.458762 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:32.458824 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:32.459450 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:32.459742 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:32.459746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:32.460415 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:32.460662 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.460668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.461547 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.461549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.462405 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.462407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.462847 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.462848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.462986 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.463616 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:32.463626 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:32.465243 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:32.467470 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:32.467490 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:32.467493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:32.467816 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:32.468310 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:32.468311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:32.468311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:32.468531 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:32.468912 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:32.468922 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:32.469073 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 07:27:32.469269 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 07:27:32.469471 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":135,"threads_started":1}
I20260810 07:27:32.470430 10594 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36423
I20260810 07:27:32.470569 10594 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:32.470773 10594 heartbeater.cc:511] Master 127.24.60.254:36423 requested a full tablet report, sending...
I20260810 07:27:32.471374 10371 ts_manager.cc:194] Registered new tserver with Master: 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173)
I20260810 07:27:32.472024 10371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:41303
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:27:32.497447 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36423
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:32.583455 10601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:32.583699 10601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:32.583729 10601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:32.585043 10601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:32.585098 10601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:32.585129 10601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:32.586807 10601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:32.587111 10601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:32.587318 10601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:32.593418 10612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.593418 10607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.593466 10608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:32.593868 10601 server_base.cc:1034] running on GCE node
I20260810 07:27:32.594115 10601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:32.594396 10601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:32.595580 10601 hybrid_clock.cc:648] HybridClock initialized: now 1786346852595580 us; error 43 us; skew 500 ppm
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:32.598498 10601 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:32.598929 10601 webserver.cc:460] Webserver started at http://127.24.60.194:42489/ using document root <none> and password file <none>
I20260810 07:27:32.599179 10601 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:32.599229 10601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:32.599381 10601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:32.601060 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "b200e7c2face24006e39be3ea14e22b8"
server_key_iv: "78ff72347b1e0e7c3ccf270dede7b319"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.601547 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "b200e7c2face24006e39be3ea14e22b8"
server_key_iv: "78ff72347b1e0e7c3ccf270dede7b319"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.603410 10601 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:32.604516 10619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.604725 10601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:32.604849 10601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "b200e7c2face24006e39be3ea14e22b8"
server_key_iv: "78ff72347b1e0e7c3ccf270dede7b319"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.604992 10601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:32.645597 10601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:32.648411 10601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:32.648613 10601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:32.648901 10601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:32.649305 10601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:32.649365 10601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.649401 10601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:32.649449 10601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.656178 10601 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:37901
I20260810 07:27:32.656273 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:37901 every 8 connection(s)
I20260810 07:27:32.656673 10601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:32.664307 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 10601
I20260810 07:27:32.664471 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:27:32.664691 24819 external_mini_cluster.cc:1428] Setting key 982acde8d0e40e2a441394148b640892
I20260810 07:27:32.667308 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.657573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:54433 (local address 127.24.60.254:36423)
0810 07:27:32.657676 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.657678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.657866 (+ 188us) server_negotiation.cc:407] Connection header received
0810 07:27:32.658394 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.658395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.658415 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.658498 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:32.659691 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.660202 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.661352 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.661587 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.661653 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.663954 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:32.663955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:32.663956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:32.663964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:32.665657 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.666165 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.666166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.666167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.666236 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.666520 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.666521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.666522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.666694 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:32.666783 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.667117 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.667211 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 07:27:32.667543 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.657490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:54433)
0810 07:27:32.657688 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:32.657734 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:32.658264 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:32.658640 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:32.658644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:32.659302 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:32.659558 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.659564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.660299 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.660300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.661218 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.661221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.661681 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.661682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.661796 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.662347 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:32.662356 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:32.663829 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:32.665745 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:32.665755 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:32.665758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:32.666075 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:32.666313 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:32.666314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:32.666314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:32.666435 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:32.666774 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:32.666782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:32.666944 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 07:27:32.667085 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 07:27:32.667272 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260810 07:27:32.668053 10759 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36423
I20260810 07:27:32.668160 10759 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:32.668390 10759 heartbeater.cc:511] Master 127.24.60.254:36423 requested a full tablet report, sending...
I20260810 07:27:32.668853 10371 ts_manager.cc:194] Registered new tserver with Master: 4d88e3fc78ff4e47ac571b71c9dcb688 (127.24.60.194:37901)
I20260810 07:27:32.669328 10371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:54433
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:27:32.695830 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36423
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:32.788261 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:32.788503 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:32.788532 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:32.789986 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:32.790040 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:32.790071 10764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:32.791855 10764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:32.792138 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:32.792398 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:32.797993 10773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.798051 10772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:32.797993 10775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:32.798272 10764 server_base.cc:1034] running on GCE node
I20260810 07:27:32.798485 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:32.798715 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:32.799865 10764 hybrid_clock.cc:648] HybridClock initialized: now 1786346852799851 us; error 34 us; skew 500 ppm
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:32.802731 10764 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:32.803260 10764 webserver.cc:460] Webserver started at http://127.24.60.195:44037/ using document root <none> and password file <none>
I20260810 07:27:32.803530 10764 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:32.803579 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:32.803722 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:32.804919 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "b25f18ea734d49879ce8753849907899"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "91fa0f1ac9270853c13c3cb063d7aec1"
server_key_iv: "16321581a2cba3924ba2205f487fc1e3"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.805339 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "b25f18ea734d49879ce8753849907899"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "91fa0f1ac9270853c13c3cb063d7aec1"
server_key_iv: "16321581a2cba3924ba2205f487fc1e3"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.806959 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:32.807796 10784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.808143 10764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:32.808257 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "b25f18ea734d49879ce8753849907899"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "91fa0f1ac9270853c13c3cb063d7aec1"
server_key_iv: "16321581a2cba3924ba2205f487fc1e3"
server_key_version: "encryptionkey@0"
I20260810 07:27:32.808347 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:32.819267 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:32.821954 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:32.822135 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:32.822419 10764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:32.822774 10764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:32.822809 10764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.822834 10764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:32.822845 10764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:32.829439 10764 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:41155
I20260810 07:27:32.829607 10923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:41155 every 8 connection(s)
I20260810 07:27:32.829923 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:32.831779 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 10764
I20260810 07:27:32.831856 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:27:32.832082 24819 external_mini_cluster.cc:1428] Setting key bbd02530e30d2279eb16169a49fd84eb
Aug 10 07:27:32 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346852, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:32.840322 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.830760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:41375 (local address 127.24.60.254:36423)
0810 07:27:32.830812 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.830814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.831140 (+ 326us) server_negotiation.cc:407] Connection header received
0810 07:27:32.831689 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.831690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.831703 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.831764 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:32.833054 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.833655 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.834722 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.834948 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.835042 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.837498 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:32.837499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:32.837500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:32.837508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:32.838758 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.839259 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.839260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.839261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.839327 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.839609 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.839610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.839611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.839747 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:32.839827 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.840133 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.840225 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 07:27:32.840531 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.830723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:41375)
0810 07:27:32.830995 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:32.831035 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:32.831579 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:32.831847 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:32.831852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:32.832689 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:32.832958 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.832964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.833767 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.833768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.834621 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.834624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.835056 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.835057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.835202 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.835796 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:32.835806 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:32.837362 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:32.838853 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:32.838861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:32.838864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:32.839165 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:32.839397 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:32.839398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:32.839398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:32.839536 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:32.839824 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:32.839832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:32.839990 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 07:27:32.840100 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 07:27:32.840278 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":106,"threads_started":1}
I20260810 07:27:32.841060 10925 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36423
I20260810 07:27:32.841202 10925 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:32.841450 10925 heartbeater.cc:511] Master 127.24.60.254:36423 requested a full tablet report, sending...
I20260810 07:27:32.841859 10371 ts_manager.cc:194] Registered new tserver with Master: b25f18ea734d49879ce8753849907899 (127.24.60.195:41155)
I20260810 07:27:32.842276 10371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:41375
I20260810 07:27:32.843885 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:32.852391 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.845297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.24.60.254:36423)
0810 07:27:32.845400 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.845403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.845522 (+ 119us) server_negotiation.cc:407] Connection header received
0810 07:27:32.845564 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.845564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.845574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.845633 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:32.846171 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.846741 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.847470 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.847675 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.847745 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.849308 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:32.849309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:32.849310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:32.849318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:32.850892 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.851331 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.851332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.851333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.851407 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:32.851660 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:32.851661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:32.851661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:32.851869 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:32.851978 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.852154 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.852250 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 07:27:32.854437 10371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:32.856341 10371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:32.863955 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.859429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:37901 (local address 127.0.0.1:46518)
0810 07:27:32.859951 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:32.859964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:32.860029 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:32.860369 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:32.860370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:32.860373 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:32.860440 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.860442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.861801 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.861802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.862869 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.862871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.863581 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.863582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.863685 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.863702 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 07:27:32.863779 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 07:27:32.863842 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260810 07:27:32.864053 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.859738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.24.60.195:41155)
0810 07:27:32.860000 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.860004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.860015 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:32.860060 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.860062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.860091 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.860405 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:32.860484 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.861325 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.862904 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.863688 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.863773 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.863834 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.863916 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.863953 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
I20260810 07:27:32.864205 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.859429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46173 (local address 127.0.0.1:37260)
0810 07:27:32.859617 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:32.859629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:32.859703 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:32.860418 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:32.860419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:32.860421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:32.860459 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.860460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.861874 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.861875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.862824 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.862827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.863987 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.863988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.864044 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.864046 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 07:27:32.864098 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 07:27:32.864139 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260810 07:27:32.864266 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.859738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.24.60.194:37901)
0810 07:27:32.859939 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.859945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.860098 (+ 153us) server_negotiation.cc:407] Connection header received
0810 07:27:32.860148 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.860150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.860165 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.860246 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:32.860524 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.861683 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.862946 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.863483 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.863578 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.863648 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.864125 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.864176 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260810 07:27:32.864452 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.860003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.24.60.193:46173)
0810 07:27:32.860233 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:32.860238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:32.860247 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:27:32.860286 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:32.860289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:32.860303 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:32.860415 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:32.860524 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.861765 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.862957 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.863688 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.863769 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.863829 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:32.864320 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 07:27:32.864373 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260810 07:27:32.864988 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:32.859427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41155 (local address 127.0.0.1:37010)
0810 07:27:32.859575 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:32.859629 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:32.859703 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:32.860205 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:32.860209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:32.860212 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:32.860321 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.860324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.861436 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.861437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.862797 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:32.862800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:32.863783 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:32.863784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:32.863840 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:32.863844 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:32.863896 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 07:27:32.863936 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":3,"thread_start_us":128,"threads_started":1}
I20260810 07:27:32.865248 10521 tablet_service.cc:1467] Processing CreateTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 (DEFAULT_TABLE table=test-table [id=5af4a35100e549f8a4b000fbe5ce6cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:32.865404 10673 tablet_service.cc:1467] Processing CreateTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 (DEFAULT_TABLE table=test-table [id=5af4a35100e549f8a4b000fbe5ce6cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:32.865716 10521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4d5e649a984d01b5a72da247cb7dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:32.865805 10673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4d5e649a984d01b5a72da247cb7dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:32.865751 10846 tablet_service.cc:1467] Processing CreateTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 (DEFAULT_TABLE table=test-table [id=5af4a35100e549f8a4b000fbe5ce6cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:32.866081 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4d5e649a984d01b5a72da247cb7dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:32.869189 10944 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Bootstrap starting.
I20260810 07:27:32.869566 10946 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Bootstrap starting.
I20260810 07:27:32.870210 10944 tablet_bootstrap.cc:654] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:32.870225 10946 tablet_bootstrap.cc:654] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:32.870872 10946 log.cc:824] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:32.870872 10944 log.cc:824] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:32.871295 10947 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Bootstrap starting.
I20260810 07:27:32.871771 10946 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: No bootstrap required, opened a new log
I20260810 07:27:32.871857 10946 ts_tablet_manager.cc:1397] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:32.872049 10947 tablet_bootstrap.cc:654] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:32.872550 10947 log.cc:824] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:32.873713 10947 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: No bootstrap required, opened a new log
I20260810 07:27:32.873802 10947 ts_tablet_manager.cc:1397] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:32.873934 10944 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: No bootstrap required, opened a new log
I20260810 07:27:32.874020 10944 ts_tablet_manager.cc:1397] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 07:27:32.874220 10946 raft_consensus.cc:348] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:32.874403 10946 raft_consensus.cc:374] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:32.874450 10946 raft_consensus.cc:729] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d88e3fc78ff4e47ac571b71c9dcb688, State: Initialized, Role: FOLLOWER
I20260810 07:27:32.874620 10946 consensus_queue.cc:260] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:32.874984 10759 heartbeater.cc:503] Master 127.24.60.254:36423 was elected leader, sending a full tablet report...
I20260810 07:27:32.874984 10946 ts_tablet_manager.cc:1428] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 07:27:32.876051 10947 raft_consensus.cc:348] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:32.876112 10944 raft_consensus.cc:348] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:32.876261 10947 raft_consensus.cc:374] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:32.876261 10944 raft_consensus.cc:374] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:32.876307 10947 raft_consensus.cc:729] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25f18ea734d49879ce8753849907899, State: Initialized, Role: FOLLOWER
I20260810 07:27:32.876307 10944 raft_consensus.cc:729] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2106bd07c93d472781a063404f6a0586, State: Initialized, Role: FOLLOWER
I20260810 07:27:32.876427 10947 consensus_queue.cc:260] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:32.876458 10944 consensus_queue.cc:260] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:32.876758 10944 ts_tablet_manager.cc:1428] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:32.876758 10947 ts_tablet_manager.cc:1428] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:32.876888 10925 heartbeater.cc:503] Master 127.24.60.254:36423 was elected leader, sending a full tablet report...
I20260810 07:27:32.876994 10594 heartbeater.cc:503] Master 127.24.60.254:36423 was elected leader, sending a full tablet report...
W20260810 07:27:32.907361 10760 tablet.cc:2369] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:27:32.958489 10597 tablet.cc:2369] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:27:33.080636 10926 tablet.cc:2369] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:33.084436 10953 raft_consensus.cc:482] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:33.084596 10953 raft_consensus.cc:504] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:33.085109 10953 leader_election.cc:288] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173), 4d88e3fc78ff4e47ac571b71c9dcb688 (127.24.60.194:37901)
I20260810 07:27:33.088534 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.085403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46173 (local address 127.24.60.195:56119)
0810 07:27:33.085489 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:33.085504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:33.085576 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:33.085859 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:33.085860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:33.085862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:33.085950 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:33.085952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.086776 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.086777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:33.087585 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:33.087587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.088185 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.088186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:33.088286 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:33.088290 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:33.088387 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 07:27:33.088429 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 07:27:33.088550 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.085531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:56119 (local address 127.24.60.193:46173)
0810 07:27:33.085605 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:33.085607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:33.085625 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:27:33.085662 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:33.085663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:33.085674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:33.085738 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:33.086059 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.086678 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.087688 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.088098 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.088157 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:33.088163 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:33.088421 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 07:27:33.088476 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:27:33.088619 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.085511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:36199 (local address 127.24.60.194:37901)
0810 07:27:33.085558 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:33.085561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:33.085855 (+ 294us) server_negotiation.cc:407] Connection header received
0810 07:27:33.085887 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:33.085889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:33.085899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:33.085959 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:33.086222 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.086709 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.087808 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.088230 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.088307 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:33.088315 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:33.088501 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 07:27:33.088545 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:27:33.088698 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.085416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:37901 (local address 127.24.60.195:36199)
0810 07:27:33.085609 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:33.085619 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:33.085693 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:33.086036 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:33.086038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:33.086041 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:33.086124 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:33.086125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.086907 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.086908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:33.087708 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:33.087711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.088320 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.088321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:33.088411 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:33.088416 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:27:33.088475 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 07:27:33.088515 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":21,"thread_start_us":102,"threads_started":1}
I20260810 07:27:33.089011 10543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d5e649a984d01b5a72da247cb7dd3" candidate_uuid: "b25f18ea734d49879ce8753849907899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2106bd07c93d472781a063404f6a0586" is_pre_election: true
I20260810 07:27:33.089130 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d5e649a984d01b5a72da247cb7dd3" candidate_uuid: "b25f18ea734d49879ce8753849907899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" is_pre_election: true
I20260810 07:27:33.089246 10543 raft_consensus.cc:2454] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b25f18ea734d49879ce8753849907899 in term 0.
I20260810 07:27:33.089334 10702 raft_consensus.cc:2454] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b25f18ea734d49879ce8753849907899 in term 0.
I20260810 07:27:33.089563 10787 leader_election.cc:302] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2106bd07c93d472781a063404f6a0586, b25f18ea734d49879ce8753849907899; no voters:
I20260810 07:27:33.089697 10953 raft_consensus.cc:2781] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:33.089757 10953 raft_consensus.cc:482] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:33.089783 10953 raft_consensus.cc:3037] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:33.090683 10953 raft_consensus.cc:504] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:33.090916 10953 leader_election.cc:288] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [CANDIDATE]: Term 1 election: Requested vote from peers 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173), 4d88e3fc78ff4e47ac571b71c9dcb688 (127.24.60.194:37901)
I20260810 07:27:33.091024 10543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d5e649a984d01b5a72da247cb7dd3" candidate_uuid: "b25f18ea734d49879ce8753849907899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2106bd07c93d472781a063404f6a0586"
I20260810 07:27:33.091068 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4d5e649a984d01b5a72da247cb7dd3" candidate_uuid: "b25f18ea734d49879ce8753849907899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
I20260810 07:27:33.091130 10543 raft_consensus.cc:3037] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:33.091187 10702 raft_consensus.cc:3037] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:33.091971 10702 raft_consensus.cc:2454] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25f18ea734d49879ce8753849907899 in term 1.
I20260810 07:27:33.092042 10543 raft_consensus.cc:2454] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25f18ea734d49879ce8753849907899 in term 1.
I20260810 07:27:33.092247 10788 leader_election.cc:302] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d88e3fc78ff4e47ac571b71c9dcb688, b25f18ea734d49879ce8753849907899; no voters:
I20260810 07:27:33.092361 10953 raft_consensus.cc:2781] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:33.092514 10953 raft_consensus.cc:686] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 1 LEADER]: Becoming Leader. State: Replica: b25f18ea734d49879ce8753849907899, State: Running, Role: LEADER
I20260810 07:27:33.092623 10953 consensus_queue.cc:237] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:33.093556 10371 catalog_manager.cc:5495] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 reported cstate change: term changed from 0 to 1, leader changed from <none> to b25f18ea734d49879ce8753849907899 (127.24.60.195). New cstate: current_term: 1 leader_uuid: "b25f18ea734d49879ce8753849907899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } health_report { overall_health: UNKNOWN } } }
I20260810 07:27:33.151391 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.148750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.24.60.195:41155)
0810 07:27:33.148827 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:33.148830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:33.148839 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:27:33.148890 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:33.148890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:33.148910 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:33.148992 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:33.149122 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.149675 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.150521 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:33.150726 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:33.150803 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:33.151007 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:33.151122 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:33.151222 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:33.151242 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 07:27:33.151268 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:27:33.153543 10543 raft_consensus.cc:1261] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Refusing update from remote peer b25f18ea734d49879ce8753849907899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:33.153543 10702 raft_consensus.cc:1261] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Refusing update from remote peer b25f18ea734d49879ce8753849907899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:33.153995 10957 consensus_queue.cc:1035] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:33.154158 10953 consensus_queue.cc:1035] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:33.157320 10962 mvcc.cc:205] Tried to move back new op lower bound from 7316876710513147904 to 7316876710267998208. Current Snapshot: MvccSnapshot[applied={T|T < 7316876710513147904}]
I20260810 07:27:33.160650 10371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:33.160781 10371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:33.161744 10371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Sending DeleteTablet for 3 replicas of tablet da4d5e649a984d01b5a72da247cb7dd3
I20260810 07:27:33.162281 10521 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:33 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:37260
I20260810 07:27:33.162385 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 10447
I20260810 07:27:33.164422 10673 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:33 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:46518
I20260810 07:27:33.169799 10846 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:33 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:37010
I20260810 07:27:33.170579 10972 tablet_replica.cc:330] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: stopping tablet replica
I20260810 07:27:33.170748 10972 raft_consensus.cc:2229] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:33.170940 10972 raft_consensus.cc:2258] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:33.171571 10972 ts_tablet_manager.cc:1905] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:33.173283 10971 tablet_replica.cc:330] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: stopping tablet replica
I20260810 07:27:33.173449 10971 raft_consensus.cc:2229] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:33.173480 10972 ts_tablet_manager.cc:1918] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:33.173563 10972 log.cc:1196] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/da4d5e649a984d01b5a72da247cb7dd3
I20260810 07:27:33.173614 10971 raft_consensus.cc:2258] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:33.173893 10972 ts_tablet_manager.cc:1939] T da4d5e649a984d01b5a72da247cb7dd3 P b25f18ea734d49879ce8753849907899: Deleting consensus metadata
I20260810 07:27:33.174357 10971 ts_tablet_manager.cc:1905] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:33.174571 10352 catalog_manager.cc:4841] TS b25f18ea734d49879ce8753849907899 (127.24.60.195:41155): tablet da4d5e649a984d01b5a72da247cb7dd3 (table test-table [id=5af4a35100e549f8a4b000fbe5ce6cb6]) successfully deleted
W20260810 07:27:33.174865 10354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:46173 (error 108)
I20260810 07:27:33.175451 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.175295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46173 (local address 127.0.0.1:37276)
0810 07:27:33.175367 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:33.175413 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 07:27:33.175594 10354 catalog_manager.cc:4584] TS 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da4d5e649a984d01b5a72da247cb7dd3: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46173: connect: Connection refused (error 111)
I20260810 07:27:33.175897 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 10601
I20260810 07:27:33.176268 10971 ts_tablet_manager.cc:1918] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:33.184557 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:33.184364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:37901 (local address 127.0.0.1:46532)
0810 07:27:33.184463 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:33.184501 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.194:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 07:27:33.184574 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 10764
I20260810 07:27:33.192783 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 10332
I20260810 07:27:53.201793 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36423
--webserver_interface=127.24.60.254
--webserver_port=33487
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:53.287597 10974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:53.287885 10974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:53.287940 10974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:53.289371 10974 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 07:27:53.289438 10974 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 07:27:53.289464 10974 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 07:27:53.289487 10974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:53.289515 10974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:53.289554 10974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:53.289587 10974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:53.289655 10974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:53.291504 10974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=33487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:53.291739 10974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:53.291936 10974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:53.302250 10980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.302284 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.302263 10983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:53.302758 10974 server_base.cc:1034] running on GCE node
I20260810 07:27:53.303100 10974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:53.303566 10974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:53.304757 10974 hybrid_clock.cc:648] HybridClock initialized: now 1786346873304742 us; error 53 us; skew 500 ppm
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:53.307783 10974 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:27:53.308321 10974 webserver.cc:460] Webserver started at http://127.24.60.254:33487/ using document root <none> and password file <none>
I20260810 07:27:53.308583 10974 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:53.308677 10974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:53.310202 10974 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:53.311085 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.311291 10974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:53.311406 10974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "d8bdb127b83545bba13d22e4f3167b4c"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "ed065666ac2a024edbe32d5094e583de"
server_key_iv: "ef27cdf664a48c11361923bf87af4159"
server_key_version: "encryptionkey@0"
I20260810 07:27:53.311862 10974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:53.323611 10974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:53.326450 10974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:53.326745 10974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:53.331503 10974 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:36423
I20260810 07:27:53.331545 11056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:36423 every 8 connection(s)
I20260810 07:27:53.331979 10974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:53.333705 11057 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:53.334367 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap starting.
I20260810 07:27:53.338296 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 10974
I20260810 07:27:53.338670 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:46173
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:36423
--webserver_port=40525
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:53.343986 11057 log.cc:824] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:53.346269 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:53.346657 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap complete.
I20260810 07:27:53.349133 11057 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:53.349472 11057 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8bdb127b83545bba13d22e4f3167b4c, State: Initialized, Role: FOLLOWER
I20260810 07:27:53.349655 11057 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:53.349777 11057 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:53.349859 11057 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:53.349912 11057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:27:53.351094 11057 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:53.351217 11057 leader_election.cc:302] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8bdb127b83545bba13d22e4f3167b4c; no voters:
I20260810 07:27:53.351434 11057 leader_election.cc:288] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:27:53.351477 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:27:53.351670 11062 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 LEADER]: Becoming Leader. State: Replica: d8bdb127b83545bba13d22e4f3167b4c, State: Running, Role: LEADER
I20260810 07:27:53.351755 11057 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:53.351847 11062 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:27:53.352200 11062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bdb127b83545bba13d22e4f3167b4c. Latest consensus state: current_term: 2 leader_uuid: "d8bdb127b83545bba13d22e4f3167b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } } }
I20260810 07:27:53.352321 11062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:53.352205 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d8bdb127b83545bba13d22e4f3167b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } } }
I20260810 07:27:53.352507 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:53.352713 11069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:53.353888 11069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:53.354038 11069 catalog_manager.cc:1259] Loaded cluster ID: 48e45ffb1f19431d8dd4eea1a107510b
I20260810 07:27:53.354085 11069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:53.355137 11069 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:53.355351 11069 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Loaded TSK: 0
I20260810 07:27:53.365505 11069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Generated new TSK 1
I20260810 07:27:53.365638 11069 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:27:53.455286 11059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:53.455540 11059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:53.455588 11059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:53.456977 11059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:53.457049 11059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:53.457101 11059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:27:53.458783 11059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:46173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=40525
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:53.459085 11059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:53.459344 11059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:53.465667 11087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.465692 11089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.465667 11086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:53.466043 11059 server_base.cc:1034] running on GCE node
I20260810 07:27:53.466274 11059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:53.466545 11059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:53.467712 11059 hybrid_clock.cc:648] HybridClock initialized: now 1786346873467700 us; error 35 us; skew 500 ppm
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:53.470749 11059 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:27:53.471380 11059 webserver.cc:460] Webserver started at http://127.24.60.193:40525/ using document root <none> and password file <none>
I20260810 07:27:53.471680 11059 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:53.471814 11059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:53.473778 11059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 07:27:53.474689 11097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.474911 11059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:53.475039 11059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "2106bd07c93d472781a063404f6a0586"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "badf5ce58093fe8680e3c46c18cea086"
server_key_iv: "86dd70271f866013d5c0179b7e3ed140"
server_key_version: "encryptionkey@0"
I20260810 07:27:53.475522 11059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:53.490643 11059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:53.493723 11059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:53.493901 11059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:53.494207 11059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:53.494875 11104 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:27:53.496640 11059 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:27:53.496716 11059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 07:27:53.496754 11059 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:27:53.497709 11059 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:27:53.497761 11059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:53.497920 11104 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Bootstrap starting.
I20260810 07:27:53.505352 11059 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46173
I20260810 07:27:53.505756 11230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46173 every 8 connection(s)
I20260810 07:27:53.505890 11059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:53.511037 11104 log.cc:824] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:53.514051 11104 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:53.514487 11104 tablet_bootstrap.cc:492] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Bootstrap complete.
I20260810 07:27:53.515913 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 11059
I20260810 07:27:53.516273 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:37901
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:36423
--webserver_port=42489
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:53.516417 11104 ts_tablet_manager.cc:1397] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:53.518853 11104 raft_consensus.cc:348] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:53.519332 11104 raft_consensus.cc:729] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2106bd07c93d472781a063404f6a0586, State: Initialized, Role: FOLLOWER
I20260810 07:27:53.519552 11104 consensus_queue.cc:260] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } }
I20260810 07:27:53.519927 11104 ts_tablet_manager.cc:1428] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:53.522301 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.506917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:51435)
0810 07:27:53.507372 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.507430 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.508362 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.508952 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.508957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.509560 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:53.509905 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.509915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.511778 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.511779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.513209 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.513213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.514072 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.514073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.514247 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.515039 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:53.515052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:53.517278 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:53.520181 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:53.520189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:53.520192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:53.520536 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:53.520852 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:53.520853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:53.520853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:53.521001 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:53.521623 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:53.521630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:53.521797 (+ 167us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.521901 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.522088 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":107,"threads_started":1}
I20260810 07:27:53.522504 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.507072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:51435 (local address 127.24.60.254:36423)
0810 07:27:53.507321 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.507332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.507579 (+ 247us) server_negotiation.cc:407] Connection header received
0810 07:27:53.508631 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.508639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.508703 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.508859 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:53.510066 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.511570 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.513305 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.513910 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.514034 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.517461 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:53.517468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:53.517468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:53.517486 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:53.520073 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.520659 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.520663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.520663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.520775 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.521085 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.521086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.521087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.521527 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:53.521626 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.522020 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.522205 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
I20260810 07:27:53.523617 11231 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36423
I20260810 07:27:53.523726 11231 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:53.523917 11231 heartbeater.cc:511] Master 127.24.60.254:36423 requested a full tablet report, sending...
I20260810 07:27:53.524854 11010 ts_manager.cc:194] Registered new tserver with Master: 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173)
I20260810 07:27:53.526463 11010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:51435
I20260810 07:27:53.527647 11231 heartbeater.cc:503] Master 127.24.60.254:36423 was elected leader, sending a full tablet report...
I20260810 07:27:53.534507 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.525731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46173 (local address 127.0.0.1:38450)
0810 07:27:53.529319 (+ 3588us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.529380 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.529482 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.529736 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.529739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.529741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:53.529833 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.529835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.531375 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.531376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.532806 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.532811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.534112 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.534114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.534228 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.534246 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.534335 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.534406 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3541,"thread_start_us":122,"threads_started":1}
I20260810 07:27:53.534933 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.526036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.24.60.193:46173)
0810 07:27:53.526232 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.526237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.529479 (+ 3242us) server_negotiation.cc:407] Connection header received
0810 07:27:53.529588 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.529592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.529614 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.529724 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:53.529918 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.531261 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.532927 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.533907 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.533995 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.534063 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.534779 (+ 716us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.534849 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260810 07:27:53.535372 11156 tablet_service.cc:1514] Processing DeleteTablet for tablet da4d5e649a984d01b5a72da247cb7dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:33 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:38450
I20260810 07:27:53.536087 11240 tablet_replica.cc:330] stopping tablet replica
I20260810 07:27:53.536217 11240 raft_consensus.cc:2229] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:53.536302 11240 raft_consensus.cc:2258] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:53.536816 11240 ts_tablet_manager.cc:1905] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:53.539973 11240 ts_tablet_manager.cc:1918] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:53.540091 11240 log.cc:1196] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/da4d5e649a984d01b5a72da247cb7dd3
I20260810 07:27:53.540393 11240 ts_tablet_manager.cc:1939] T da4d5e649a984d01b5a72da247cb7dd3 P 2106bd07c93d472781a063404f6a0586: Deleting consensus metadata
I20260810 07:27:53.541138 10994 catalog_manager.cc:4841] TS 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173): tablet da4d5e649a984d01b5a72da247cb7dd3 (table test-table [id=5af4a35100e549f8a4b000fbe5ce6cb6]) successfully deleted
W20260810 07:27:53.619958 11236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:53.620297 11236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:53.620354 11236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:53.621922 11236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:53.622005 11236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:53.622061 11236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:27:53.623956 11236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:37901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=42489
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:53.624326 11236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:53.624619 11236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:53.630869 11249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.630861 11245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.630861 11246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:53.631125 11236 server_base.cc:1034] running on GCE node
I20260810 07:27:53.631415 11236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:53.631706 11236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:53.632881 11236 hybrid_clock.cc:648] HybridClock initialized: now 1786346873632863 us; error 42 us; skew 500 ppm
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:53.635876 11236 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:27:53.636430 11236 webserver.cc:460] Webserver started at http://127.24.60.194:42489/ using document root <none> and password file <none>
I20260810 07:27:53.636775 11236 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:53.636869 11236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:53.639055 11236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:53.639941 11258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.640142 11236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:53.640250 11236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "b200e7c2face24006e39be3ea14e22b8"
server_key_iv: "78ff72347b1e0e7c3ccf270dede7b319"
server_key_version: "encryptionkey@0"
I20260810 07:27:53.640635 11236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:53.651741 11236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:53.654239 11236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:53.654418 11236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:53.654670 11236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:53.655140 11267 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260810 07:27:53.656654 11267 ts_tablet_manager.cc:1852] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 07:27:53.656975 11267 ts_tablet_manager.cc:1905] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:53.659574 11267 ts_tablet_manager.cc:1918] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:53.659709 11267 log.cc:1196] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/da4d5e649a984d01b5a72da247cb7dd3
I20260810 07:27:53.660064 11267 ts_tablet_manager.cc:1939] T da4d5e649a984d01b5a72da247cb7dd3 P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting consensus metadata
I20260810 07:27:53.660332 11236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 07:27:53.660392 11236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 07:27:53.660449 11236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:53.660497 11236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.667218 11236 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:37901
I20260810 07:27:53.667300 11397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:37901 every 8 connection(s)
I20260810 07:27:53.667681 11236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:53.677382 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 11236
I20260810 07:27:53.677753 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:41155
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:36423
--webserver_port=44037
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:53.677800 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.668529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:51311 (local address 127.24.60.254:36423)
0810 07:27:53.668592 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.668594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.668852 (+ 258us) server_negotiation.cc:407] Connection header received
0810 07:27:53.669558 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.669560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.669576 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.669634 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:53.670329 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.670844 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.671974 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.672192 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.672278 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.674792 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:53.674794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:53.674794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:53.674809 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:53.676119 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.676552 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.676553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.676554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.676619 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.676893 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.676894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.676895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.677027 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:53.677100 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.677599 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.677722 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 07:27:53.678012 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.668528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:51311)
0810 07:27:53.668723 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.668762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.669432 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.669721 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.669724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.670015 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:53.670222 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.670227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.670948 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.670949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.671879 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.671882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.672290 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.672291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.672430 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.673122 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:53.673131 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:53.674683 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:53.676211 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:53.676221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:53.676223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:53.676466 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:53.676689 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:53.676690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:53.676690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:53.676818 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:53.677103 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:53.677112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:53.677340 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.677480 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.677703 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260810 07:27:53.678570 11398 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36423
I20260810 07:27:53.678694 11398 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:53.678913 11398 heartbeater.cc:511] Master 127.24.60.254:36423 requested a full tablet report, sending...
I20260810 07:27:53.679411 11010 ts_manager.cc:194] Registered new tserver with Master: 4d88e3fc78ff4e47ac571b71c9dcb688 (127.24.60.194:37901)
I20260810 07:27:53.679863 11010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:51311
W20260810 07:27:53.765072 11401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:53.765332 11401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:53.765362 11401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:53.766547 11401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:53.766588 11401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:53.766615 11401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:27:53.768082 11401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:41155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=44037
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:27:53.768303 11401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:53.768504 11401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:53.774201 11412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.774256 11410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.774204 11409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:53.774490 11401 server_base.cc:1034] running on GCE node
I20260810 07:27:53.774725 11401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:53.775007 11401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:53.776185 11401 hybrid_clock.cc:648] HybridClock initialized: now 1786346873776164 us; error 37 us; skew 500 ppm
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:53.779047 11401 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:27:53.779651 11401 webserver.cc:460] Webserver started at http://127.24.60.195:44037/ using document root <none> and password file <none>
I20260810 07:27:53.779894 11401 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:53.779997 11401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:53.781678 11401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:27:53.782363 11421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.782575 11401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:27:53.782686 11401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "b25f18ea734d49879ce8753849907899"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "91fa0f1ac9270853c13c3cb063d7aec1"
server_key_iv: "16321581a2cba3924ba2205f487fc1e3"
server_key_version: "encryptionkey@0"
I20260810 07:27:53.783074 11401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:53.792500 11401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:53.795094 11401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:53.795274 11401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:53.795549 11401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:53.795881 11401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:53.795933 11401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.795974 11401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:53.796005 11401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.805627 11401 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:41155
I20260810 07:27:53.805680 11560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:41155 every 8 connection(s)
I20260810 07:27:53.806077 11401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:27:53 dist-test-slave-vxj2 krb5kdc[10318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:27:53.814946 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 11401
I20260810 07:27:53.817085 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.806911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:55981 (local address 127.24.60.254:36423)
0810 07:27:53.806982 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.806984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.807213 (+ 229us) server_negotiation.cc:407] Connection header received
0810 07:27:53.807785 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.807787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.807800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.807847 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:53.808876 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.809453 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.810457 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.810676 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.810760 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.813358 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:53.813360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:53.813361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:53.813368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:53.815342 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.815864 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.815866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.815866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.815947 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.816314 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.816315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.816315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.816441 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:53.816503 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.816890 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.816984 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:27:53.817207 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.806895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:55981)
0810 07:27:53.807086 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.807126 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.807667 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.807933 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.807937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.808551 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:53.808758 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.808763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.809550 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.809551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.810363 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.810365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.810769 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.810771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.810933 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.811487 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:53.811496 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:53.813252 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:53.815455 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:53.815467 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:53.815470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:53.815789 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:53.816034 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:53.816035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:53.816036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:53.816228 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:53.816513 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:53.816522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:53.816732 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.816826 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.816971 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260810 07:27:53.817756 11561 heartbeater.cc:347] Connected to a master server at 127.24.60.254:36423
I20260810 07:27:53.817876 11561 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:53.818073 11561 heartbeater.cc:511] Master 127.24.60.254:36423 requested a full tablet report, sending...
I20260810 07:27:53.818555 11010 ts_manager.cc:194] Registered new tserver with Master: b25f18ea734d49879ce8753849907899 (127.24.60.195:41155)
I20260810 07:27:53.819166 11010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:55981
I20260810 07:27:53.823769 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.815415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.24.60.254:36423)
0810 07:27:53.815538 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.815540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.815682 (+ 142us) server_negotiation.cc:407] Connection header received
0810 07:27:53.815771 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.815772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.815793 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.815894 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:53.816708 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.817519 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.818596 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.818916 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.819001 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.820620 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:53.820622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:53.820622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:53.820629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:53.822583 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.823040 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.823041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.823042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.823101 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.823328 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.823329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.823329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.823448 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:53.823590 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.823623 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.823694 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260810 07:27:53.824579 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:53.827584 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.824931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.24.60.254:36423)
0810 07:27:53.824991 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.824993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.825289 (+ 296us) server_negotiation.cc:407] Connection header received
0810 07:27:53.825386 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.825387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.825425 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.825488 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:53.825611 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.826103 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.826888 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.827088 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.827145 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.827271 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:53.827326 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 07:27:53.827510 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35722: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 07:27:53.827663 11567 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35722: token expired
W20260810 07:27:53.827675 10928 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.60.254:36423)
I20260810 07:27:53.833832 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.827962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.24.60.254:36423)
0810 07:27:53.828005 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.828007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.828014 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:27:53.828032 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.828033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.828041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.828089 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:53.828596 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.829064 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.829812 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.830015 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.830090 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.831284 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:53.831286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:53.831286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:53.831294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:53.832488 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.832840 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.832841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.832842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.832956 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:53.833202 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:53.833203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:53.833204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:53.833376 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:53.833485 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.833626 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.833723 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:27:53.835081 11010 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:53.836755 11010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:53.839115 11156 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0dd51b6316482d84a72a94b55757bc (DEFAULT_TABLE table=test-table [id=5042973ccd1d493a941c2d405fd29b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:53.839359 11156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0dd51b6316482d84a72a94b55757bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:53.841053 11104 tablet_bootstrap.cc:492] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Bootstrap starting.
I20260810 07:27:53.841424 11104 tablet_bootstrap.cc:654] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:53.842034 11104 tablet_bootstrap.cc:492] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: No bootstrap required, opened a new log
I20260810 07:27:53.842092 11104 ts_tablet_manager.cc:1397] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:27:53.842217 11104 raft_consensus.cc:348] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.842300 11104 raft_consensus.cc:374] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:53.842362 11104 raft_consensus.cc:729] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2106bd07c93d472781a063404f6a0586, State: Initialized, Role: FOLLOWER
I20260810 07:27:53.842427 11104 consensus_queue.cc:260] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.842536 11104 ts_tablet_manager.cc:1428] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 07:27:53.843392 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.838901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.24.60.194:37901)
0810 07:27:53.839302 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.839307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.839320 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:53.839362 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.839364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.839388 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.839540 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:53.839665 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.840746 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.841995 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.843019 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.843118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.843184 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.843262 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.843301 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":92,"threads_started":1}
I20260810 07:27:53.843443 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.838767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:37901 (local address 127.0.0.1:34082)
0810 07:27:53.838824 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.838837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.838878 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.839544 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.839545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.839547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:53.839597 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.839599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.840839 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.840840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.841898 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.841902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.843116 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.843117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.843179 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.843183 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.843246 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.843301 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 07:27:53.843451 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.838825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41155 (local address 127.0.0.1:32854)
0810 07:27:53.839009 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.839019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.839065 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.839592 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.839592 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.839594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:53.839630 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.839631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.840708 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.840709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.842158 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.842160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.843204 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.843205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.843270 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.843275 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.843307 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.843342 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260810 07:27:53.843722 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.839043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.24.60.195:41155)
0810 07:27:53.839297 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.839303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.839314 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:27:53.839357 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.839359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.839377 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.839483 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:53.839710 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.840615 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.842282 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.843085 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.843472 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.843552 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.843583 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.843638 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
I20260810 07:27:53.844148 11318 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0dd51b6316482d84a72a94b55757bc (DEFAULT_TABLE table=test-table [id=5042973ccd1d493a941c2d405fd29b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:53.844364 11482 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0dd51b6316482d84a72a94b55757bc (DEFAULT_TABLE table=test-table [id=5042973ccd1d493a941c2d405fd29b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:53.844523 11318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0dd51b6316482d84a72a94b55757bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:53.844764 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0dd51b6316482d84a72a94b55757bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:53.846578 11267 tablet_bootstrap.cc:492] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Bootstrap starting.
I20260810 07:27:53.847337 11267 tablet_bootstrap.cc:654] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:53.847652 11267 log.cc:824] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:53.847908 11574 tablet_bootstrap.cc:492] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Bootstrap starting.
I20260810 07:27:53.848587 11267 tablet_bootstrap.cc:492] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: No bootstrap required, opened a new log
I20260810 07:27:53.848658 11267 ts_tablet_manager.cc:1397] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:27:53.848783 11574 tablet_bootstrap.cc:654] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:53.849475 11574 log.cc:824] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:53.850329 11574 tablet_bootstrap.cc:492] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: No bootstrap required, opened a new log
I20260810 07:27:53.850411 11574 ts_tablet_manager.cc:1397] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 07:27:53.850989 11267 raft_consensus.cc:348] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.851188 11267 raft_consensus.cc:374] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:53.851228 11267 raft_consensus.cc:729] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d88e3fc78ff4e47ac571b71c9dcb688, State: Initialized, Role: FOLLOWER
I20260810 07:27:53.851366 11267 consensus_queue.cc:260] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.851683 11267 ts_tablet_manager.cc:1428] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:53.851823 11398 heartbeater.cc:503] Master 127.24.60.254:36423 was elected leader, sending a full tablet report...
I20260810 07:27:53.852751 11574 raft_consensus.cc:348] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.853070 11574 raft_consensus.cc:374] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:53.853135 11574 raft_consensus.cc:729] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25f18ea734d49879ce8753849907899, State: Initialized, Role: FOLLOWER
I20260810 07:27:53.853292 11574 consensus_queue.cc:260] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.853612 11574 ts_tablet_manager.cc:1428] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 07:27:53.853623 11561 heartbeater.cc:503] Master 127.24.60.254:36423 was elected leader, sending a full tablet report...
W20260810 07:27:53.918330 11399 tablet.cc:2369] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:53.954201 11237 raft_consensus.cc:482] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:53.954345 11237 raft_consensus.cc:504] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.954694 11237 leader_election.cc:288] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d88e3fc78ff4e47ac571b71c9dcb688 (127.24.60.194:37901), b25f18ea734d49879ce8753849907899 (127.24.60.195:41155)
I20260810 07:27:53.958225 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.954897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:37901 (local address 127.24.60.193:41605)
0810 07:27:53.954988 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.955002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.955053 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.955224 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.955225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.955228 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:53.955308 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.955310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.956011 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.956012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.956997 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.957000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.957906 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.957907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.957980 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.957984 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.958078 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.958120 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 07:27:53.958325 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.955018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:36311 (local address 127.24.60.195:41155)
0810 07:27:53.955078 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.955080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.955356 (+ 276us) server_negotiation.cc:407] Connection header received
0810 07:27:53.955420 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.955422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.955431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.955485 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:53.955656 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.956357 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.957405 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.957963 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.958021 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.958027 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.958210 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.958256 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:27:53.958389 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.954902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41155 (local address 127.24.60.193:36311)
0810 07:27:53.955276 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.955286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:53.955361 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:53.955487 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:53.955488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:53.955489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:53.955586 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.955588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.956451 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.956452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.957323 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:53.957326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.958077 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.958078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:53.958136 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.958139 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:27:53.958195 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 07:27:53.958224 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":207,"thread_start_us":109,"threads_started":1}
I20260810 07:27:53.958496 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.954987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:41605 (local address 127.24.60.194:37901)
0810 07:27:53.955047 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.955049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.955062 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:53.955123 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.955124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.955139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.955216 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:53.955392 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.955925 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.957094 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.957755 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.957836 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.957845 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.958366 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.958422 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 07:27:53.958877 11343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0dd51b6316482d84a72a94b55757bc" candidate_uuid: "2106bd07c93d472781a063404f6a0586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" is_pre_election: true
I20260810 07:27:53.958917 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0dd51b6316482d84a72a94b55757bc" candidate_uuid: "2106bd07c93d472781a063404f6a0586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25f18ea734d49879ce8753849907899" is_pre_election: true
I20260810 07:27:53.959041 11343 raft_consensus.cc:2454] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2106bd07c93d472781a063404f6a0586 in term 0.
I20260810 07:27:53.959082 11507 raft_consensus.cc:2454] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2106bd07c93d472781a063404f6a0586 in term 0.
I20260810 07:27:53.959281 11101 leader_election.cc:302] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2106bd07c93d472781a063404f6a0586, 4d88e3fc78ff4e47ac571b71c9dcb688; no voters:
I20260810 07:27:53.959430 11237 raft_consensus.cc:2781] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:53.959494 11237 raft_consensus.cc:482] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:53.959517 11237 raft_consensus.cc:3037] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:53.960297 11237 raft_consensus.cc:504] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.960477 11237 leader_election.cc:288] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [CANDIDATE]: Term 1 election: Requested vote from peers 4d88e3fc78ff4e47ac571b71c9dcb688 (127.24.60.194:37901), b25f18ea734d49879ce8753849907899 (127.24.60.195:41155)
I20260810 07:27:53.960672 11507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0dd51b6316482d84a72a94b55757bc" candidate_uuid: "2106bd07c93d472781a063404f6a0586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25f18ea734d49879ce8753849907899"
I20260810 07:27:53.960769 11507 raft_consensus.cc:3037] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:53.961622 11507 raft_consensus.cc:2454] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2106bd07c93d472781a063404f6a0586 in term 1.
I20260810 07:27:53.961784 11343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0dd51b6316482d84a72a94b55757bc" candidate_uuid: "2106bd07c93d472781a063404f6a0586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
I20260810 07:27:53.961848 11099 leader_election.cc:302] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2106bd07c93d472781a063404f6a0586, b25f18ea734d49879ce8753849907899; no voters:
I20260810 07:27:53.961882 11343 raft_consensus.cc:3037] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:53.961941 11237 raft_consensus.cc:2781] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:53.962096 11237 raft_consensus.cc:686] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 1 LEADER]: Becoming Leader. State: Replica: 2106bd07c93d472781a063404f6a0586, State: Running, Role: LEADER
I20260810 07:27:53.962178 11237 consensus_queue.cc:237] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } }
I20260810 07:27:53.962879 11343 raft_consensus.cc:2454] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2106bd07c93d472781a063404f6a0586 in term 1.
I20260810 07:27:53.963109 11010 catalog_manager.cc:5495] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2106bd07c93d472781a063404f6a0586 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "2106bd07c93d472781a063404f6a0586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2106bd07c93d472781a063404f6a0586" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260810 07:27:53.990200 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.987898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.24.60.193:46173)
0810 07:27:53.987984 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:53.987986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:53.988001 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:27:53.988043 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:53.988043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:53.988063 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:53.988141 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:53.988266 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.988743 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.989509 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:53.989708 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:53.989772 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:53.989906 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:53.990020 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:53.990081 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:53.990113 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 07:27:53.990136 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:27:53.991859 11507 raft_consensus.cc:1261] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Refusing update from remote peer 2106bd07c93d472781a063404f6a0586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:53.991859 11343 raft_consensus.cc:1261] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Refusing update from remote peer 2106bd07c93d472781a063404f6a0586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:53.992270 11237 consensus_queue.cc:1035] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d88e3fc78ff4e47ac571b71c9dcb688" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:53.992480 11580 consensus_queue.cc:1035] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25f18ea734d49879ce8753849907899" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:53.995271 11583 mvcc.cc:205] Tried to move back new op lower bound from 7316876795867717632 to 7316876795749687296. Current Snapshot: MvccSnapshot[applied={T|T < 7316876795867717632}]
I20260810 07:27:53.995817 11586 mvcc.cc:205] Tried to move back new op lower bound from 7316876795867717632 to 7316876795749687296. Current Snapshot: MvccSnapshot[applied={T|T < 7316876795867717632}]
I20260810 07:27:53.997072 11584 mvcc.cc:205] Tried to move back new op lower bound from 7316876795867717632 to 7316876795749687296. Current Snapshot: MvccSnapshot[applied={T|T < 7316876795867717632 or (T in {7316876795867717632})}]
I20260810 07:27:53.998395 11007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:53.998512 11007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:53.999238 11007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Sending DeleteTablet for 3 replicas of tablet 8d0dd51b6316482d84a72a94b55757bc
I20260810 07:27:53.999588 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0dd51b6316482d84a72a94b55757bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:53 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:32854
I20260810 07:27:53.999612 11156 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0dd51b6316482d84a72a94b55757bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:53 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:38450
I20260810 07:27:53.999773 11318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0dd51b6316482d84a72a94b55757bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:53 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:34082
I20260810 07:27:54.000015 11240 tablet_replica.cc:330] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: stopping tablet replica
I20260810 07:27:54.000059 11598 tablet_replica.cc:330] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: stopping tablet replica
I20260810 07:27:54.000090 11240 raft_consensus.cc:2229] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:54.000162 11598 raft_consensus.cc:2229] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:54.000185 11240 raft_consensus.cc:2258] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:54.000288 11598 raft_consensus.cc:2258] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:54.000599 11240 ts_tablet_manager.cc:1905] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:54.000725 11598 ts_tablet_manager.cc:1905] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:54.002494 11597 tablet_replica.cc:330] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: stopping tablet replica
I20260810 07:27:54.002640 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 11059
I20260810 07:27:54.002662 11597 raft_consensus.cc:2229] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:54.008919 11597 raft_consensus.cc:2258] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:54.013720 11597 ts_tablet_manager.cc:1905] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:54.014081 11598 ts_tablet_manager.cc:1918] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:54.014107 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 11236
W20260810 07:27:54.014099 10994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:46173 (error 108)
I20260810 07:27:54.014269 11598 log.cc:1196] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/8d0dd51b6316482d84a72a94b55757bc
I20260810 07:27:54.015985 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.015823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46173 (local address 127.0.0.1:38460)
0810 07:27:54.015881 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.015932 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 07:27:54.016273 10994 catalog_manager.cc:4584] TS 2106bd07c93d472781a063404f6a0586 (127.24.60.193:46173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d0dd51b6316482d84a72a94b55757bc: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46173: connect: Connection refused (error 111)
I20260810 07:27:54.017128 11597 ts_tablet_manager.cc:1918] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:54.017247 11597 log.cc:1196] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/8d0dd51b6316482d84a72a94b55757bc
I20260810 07:27:54.017717 11597 ts_tablet_manager.cc:1939] T 8d0dd51b6316482d84a72a94b55757bc P b25f18ea734d49879ce8753849907899: Deleting consensus metadata
I20260810 07:27:54.018273 10993 catalog_manager.cc:4841] TS b25f18ea734d49879ce8753849907899 (127.24.60.195:41155): tablet 8d0dd51b6316482d84a72a94b55757bc (table test-table [id=5042973ccd1d493a941c2d405fd29b4c]) successfully deleted
I20260810 07:27:54.021540 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 11401
I20260810 07:27:54.021608 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.021400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:37901 (local address 127.0.0.1:34088)
0810 07:27:54.021508 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.021559 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.194:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 07:27:54.029090 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 10974
I20260810 07:28:14.039498 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36423
--webserver_interface=127.24.60.254
--webserver_port=33487
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:14.126549 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:14.126784 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:14.126811 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:14.128053 11600 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 07:28:14.128096 11600 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 07:28:14.128109 11600 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 07:28:14.128121 11600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:14.128134 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:14.128147 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:14.128159 11600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:14.128171 11600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:14.129770 11600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=33487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:14.130002 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:14.130201 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:28:14.135892 11600 server_base.cc:1034] running on GCE node
W20260810 07:28:14.135936 11606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.136066 11608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.135958 11605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:14.136443 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:14.136804 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:14.137982 11600 hybrid_clock.cc:648] HybridClock initialized: now 1786346894137963 us; error 40 us; skew 500 ppm
Aug 10 07:28:14 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346894, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:14.140961 11600 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:28:14.141538 11600 webserver.cc:460] Webserver started at http://127.24.60.254:33487/ using document root <none> and password file <none>
I20260810 07:28:14.141819 11600 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:14.141896 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:14.143430 11600 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:28:14.144403 11616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.144645 11600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:14.144770 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "d8bdb127b83545bba13d22e4f3167b4c"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "ed065666ac2a024edbe32d5094e583de"
server_key_iv: "ef27cdf664a48c11361923bf87af4159"
server_key_version: "encryptionkey@0"
I20260810 07:28:14.145361 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:14.158985 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:14.161253 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:14.161440 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:14.165958 11600 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:36423
I20260810 07:28:14.166003 11676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:36423 every 8 connection(s)
I20260810 07:28:14.166452 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:14.168213 11679 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:28:14.169028 11679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap starting.
I20260810 07:28:14.176153 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 11600
I20260810 07:28:14.176608 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:46173
--local_ip_for_outbound_sockets=127.24.60.193
--tserver_master_addrs=127.24.60.254:36423
--webserver_port=40525
--webserver_interface=127.24.60.193
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:28:14.178385 11679 log.cc:824] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:14.181010 11679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:28:14.181408 11679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Bootstrap complete.
I20260810 07:28:14.183876 11679 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:28:14.184149 11679 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8bdb127b83545bba13d22e4f3167b4c, State: Initialized, Role: FOLLOWER
I20260810 07:28:14.184365 11679 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:28:14.184476 11679 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:14.184547 11679 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:14.184603 11679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:28:14.185808 11679 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:28:14.185952 11679 leader_election.cc:302] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8bdb127b83545bba13d22e4f3167b4c; no voters:
I20260810 07:28:14.186200 11679 leader_election.cc:288] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 07:28:14.186250 11683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 3 FOLLOWER]: Leader election won for term 3
I20260810 07:28:14.186398 11683 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [term 3 LEADER]: Becoming Leader. State: Replica: d8bdb127b83545bba13d22e4f3167b4c, State: Running, Role: LEADER
I20260810 07:28:14.186522 11679 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:14.186492 11683 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } }
I20260810 07:28:14.186827 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bdb127b83545bba13d22e4f3167b4c. Latest consensus state: current_term: 3 leader_uuid: "d8bdb127b83545bba13d22e4f3167b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } } }
I20260810 07:28:14.186954 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:14.186818 11684 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d8bdb127b83545bba13d22e4f3167b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bdb127b83545bba13d22e4f3167b4c" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 36423 } } }
I20260810 07:28:14.187067 11684 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:14.187544 11688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:14.188570 11688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:14.188723 11688 catalog_manager.cc:1259] Loaded cluster ID: 48e45ffb1f19431d8dd4eea1a107510b
I20260810 07:28:14.189002 11688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:14.189949 11688 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:14.190167 11688 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Loaded TSK: 0
I20260810 07:28:14.190229 11688 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8bdb127b83545bba13d22e4f3167b4c: Loaded TSK: 1
F20260810 07:28:14.190470 11688 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4c923b99ad google::LogMessage::Fail() at ??:0
@ 0x7f4c923bc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4c923b94de google::LogMessage::Flush() at ??:0
@ 0x7f4c923baa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4c94b26104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4c94b23d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4c9a3131cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4c9a31348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4c9a313594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4c9a2d9c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4c9a2e3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4c942870d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4c9427c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4c950e06db start_thread at ??:0
@ 0x7f4c90eb761f clone at ??:0
W20260810 07:28:14.283871 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:14.284147 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:14.284200 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:14.285573 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:14.285651 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:14.285698 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:14.287374 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:46173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=40525
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:14.287667 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:14.287955 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:14.293581 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.293593 11726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:14.293927 11681 server_base.cc:1034] running on GCE node
W20260810 07:28:14.293557 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:14.294241 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:14.294533 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:14.295734 11681 hybrid_clock.cc:648] HybridClock initialized: now 1786346894295708 us; error 45 us; skew 500 ppm
Aug 10 07:28:14 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346894, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:14.298647 11681 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:28:14.299189 11681 webserver.cc:460] Webserver started at http://127.24.60.193:40525/ using document root <none> and password file <none>
I20260810 07:28:14.299464 11681 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:14.299564 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:14.301146 11681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:14.302059 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.302273 11681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:14.302402 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "2106bd07c93d472781a063404f6a0586"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "badf5ce58093fe8680e3c46c18cea086"
server_key_iv: "86dd70271f866013d5c0179b7e3ed140"
server_key_version: "encryptionkey@0"
I20260810 07:28:14.302829 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:14.312480 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:14.314986 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:14.315171 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:14.315438 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:14.315897 11745 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260810 07:28:14.317979 11745 ts_tablet_manager.cc:1852] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 07:28:14.318383 11745 ts_tablet_manager.cc:1905] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:14.321264 11745 ts_tablet_manager.cc:1918] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:14.321395 11745 log.cc:1196] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/wal/wals/8d0dd51b6316482d84a72a94b55757bc
I20260810 07:28:14.321719 11745 ts_tablet_manager.cc:1939] T 8d0dd51b6316482d84a72a94b55757bc P 2106bd07c93d472781a063404f6a0586: Deleting consensus metadata
I20260810 07:28:14.321993 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 07:28:14.322086 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 07:28:14.322141 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:14.322186 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.329600 11681 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46173
I20260810 07:28:14.329692 11865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46173 every 8 connection(s)
I20260810 07:28:14.330060 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:14.333434 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 11681
I20260810 07:28:14.333801 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:37901
--local_ip_for_outbound_sockets=127.24.60.194
--tserver_master_addrs=127.24.60.254:36423
--webserver_port=42489
--webserver_interface=127.24.60.194
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:14.412609 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:28:14.330906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:34673)
0810 07:28:14.331113 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.331153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:14.331750 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:14.412363 (+ 80613us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":113,"threads_started":1}
W20260810 07:28:14.412938 11866 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:36423 (0 consecutive failures): Network error: Failed to ping master at 127.24.60.254:36423: Client connection negotiation failed: client connection to 127.24.60.254:36423: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:28:14.413396 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.413241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:52921)
0810 07:28:14.413299 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.413355 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 07:28:14.413831 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.413717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:49177)
0810 07:28:14.413771 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.413805 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 07:28:14.413968 11866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:28:14.425419 11869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:14.425657 11869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:14.425685 11869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:14.426930 11869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:14.426975 11869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:14.427004 11869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:14.428520 11869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:37901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=42489
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:14.428735 11869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:14.428946 11869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:14.434532 11875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.434499 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.434494 11878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:14.434811 11869 server_base.cc:1034] running on GCE node
I20260810 07:28:14.435043 11869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:14.435277 11869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:14.436465 11869 hybrid_clock.cc:648] HybridClock initialized: now 1786346894436457 us; error 38 us; skew 500 ppm
Aug 10 07:28:14 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346894, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:14.439373 11869 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:28:14.439886 11869 webserver.cc:460] Webserver started at http://127.24.60.194:42489/ using document root <none> and password file <none>
I20260810 07:28:14.440088 11869 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:14.440131 11869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:14.441656 11869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:14.442379 11887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.442674 11869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:14.442802 11869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "4d88e3fc78ff4e47ac571b71c9dcb688"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "b200e7c2face24006e39be3ea14e22b8"
server_key_iv: "78ff72347b1e0e7c3ccf270dede7b319"
server_key_version: "encryptionkey@0"
I20260810 07:28:14.443219 11869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:14.459425 11869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:14.462141 11869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:14.462314 11869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:14.462594 11869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:14.463091 11895 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260810 07:28:14.464640 11895 ts_tablet_manager.cc:1852] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 07:28:14.464929 11895 ts_tablet_manager.cc:1905] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:14.467450 11895 ts_tablet_manager.cc:1918] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:14.467561 11895 ts_tablet_manager.cc:1939] T 8d0dd51b6316482d84a72a94b55757bc P 4d88e3fc78ff4e47ac571b71c9dcb688: Deleting consensus metadata
I20260810 07:28:14.467761 11869 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 07:28:14.467819 11869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 07:28:14.467849 11869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:14.467861 11869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.474684 11869 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:37901
I20260810 07:28:14.474854 12021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:37901 every 8 connection(s)
I20260810 07:28:14.475138 11869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:14.476385 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.475921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:51345)
0810 07:28:14.476124 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.476206 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
W20260810 07:28:14.476624 12022 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:36423 (0 consecutive failures): Network error: Failed to ping master at 127.24.60.254:36423: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
I20260810 07:28:14.476974 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.476876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:52799)
0810 07:28:14.476927 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.476955 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 07:28:14.477284 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.477214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:52843)
0810 07:28:14.477246 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.477266 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260810 07:28:14.477384 12022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:28:14.481217 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 11869
I20260810 07:28:14.481562 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:41155
--local_ip_for_outbound_sockets=127.24.60.195
--tserver_master_addrs=127.24.60.254:36423
--webserver_port=44037
--webserver_interface=127.24.60.195
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:14.569743 12025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:14.569972 12025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:14.569999 12025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:14.571260 12025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:14.571313 12025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:14.571342 12025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:14.572924 12025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:41155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=44037
--tserver_master_addrs=127.24.60.254:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:14.573247 12025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:14.573535 12025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:14.579244 12032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.579223 12034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:14.579227 12031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:14.579515 12025 server_base.cc:1034] running on GCE node
I20260810 07:28:14.579727 12025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:14.579960 12025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:14.581112 12025 hybrid_clock.cc:648] HybridClock initialized: now 1786346894581100 us; error 29 us; skew 500 ppm
Aug 10 07:28:14 dist-test-slave-vxj2 krb5kdc[10318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346894, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:14.584233 12025 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:28:14.584795 12025 webserver.cc:460] Webserver started at http://127.24.60.195:44037/ using document root <none> and password file <none>
I20260810 07:28:14.585079 12025 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:14.585183 12025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:14.586773 12025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:14.587589 12041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.587786 12025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:14.587903 12025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "b25f18ea734d49879ce8753849907899"
format_stamp: "Formatted at 2026-08-10 07:27:32 on dist-test-slave-vxj2"
server_key: "91fa0f1ac9270853c13c3cb063d7aec1"
server_key_iv: "16321581a2cba3924ba2205f487fc1e3"
server_key_version: "encryptionkey@0"
I20260810 07:28:14.588299 12025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:14.610241 12025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:14.613081 12025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:14.613300 12025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:14.613603 12025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:14.613982 12025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:14.614048 12025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.614094 12025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:14.614142 12025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:14.620941 12025 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:41155
I20260810 07:28:14.620997 12172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:41155 every 8 connection(s)
I20260810 07:28:14.621533 12025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:14.622648 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.622241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:40777)
0810 07:28:14.622408 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.622493 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
W20260810 07:28:14.623116 12173 heartbeater.cc:644] Failed to heartbeat to 127.24.60.254:36423 (0 consecutive failures): Network error: Failed to ping master at 127.24.60.254:36423: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
I20260810 07:28:14.623461 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.623353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:33265)
0810 07:28:14.623406 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.623440 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 07:28:14.623744 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:14.623654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:47443)
0810 07:28:14.623699 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:14.623723 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 07:28:14.623862 12173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:28:14.628046 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12025
I20260810 07:28:15.414763 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:15.414451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:51387)
0810 07:28:15.414654 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:15.414694 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":136,"threads_started":1}
I20260810 07:28:15.478076 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:15.477783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:37557)
0810 07:28:15.477960 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:15.478015 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260810 07:28:15.624431 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:15.624197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:51515)
0810 07:28:15.624329 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:15.624371 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260810 07:28:16.415658 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:16.415428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:53725)
0810 07:28:16.415570 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:16.415608 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260810 07:28:16.479247 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:16.478973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:33029)
0810 07:28:16.479141 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:16.479200 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260810 07:28:16.625285 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:16.625015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:45219)
0810 07:28:16.625185 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:16.625233 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260810 07:28:17.416545 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:17.416257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:35533)
0810 07:28:17.416449 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:17.416496 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":122,"threads_started":1}
I20260810 07:28:17.480168 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:17.479883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:52295)
0810 07:28:17.480069 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:17.480119 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260810 07:28:17.626156 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:17.625904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:37775)
0810 07:28:17.626059 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:17.626107 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260810 07:28:18.417554 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:18.417241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:52009)
0810 07:28:18.417411 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:18.417505 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260810 07:28:18.481050 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:18.480791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:36519)
0810 07:28:18.480944 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:18.481001 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260810 07:28:18.627069 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:18.626755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:34593)
0810 07:28:18.626966 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:18.627018 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":68,"threads_started":1}
I20260810 07:28:19.418393 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:19.418138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:34821)
0810 07:28:19.418298 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:19.418346 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260810 07:28:19.481948 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:19.481665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:40227)
0810 07:28:19.481810 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:19.481859 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260810 07:28:19.627905 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:19.627664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:60269)
0810 07:28:19.627812 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:19.627853 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260810 07:28:20.419296 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:20.419021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:46205)
0810 07:28:20.419197 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:20.419245 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260810 07:28:20.482769 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:20.482532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:59529)
0810 07:28:20.482676 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:20.482722 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260810 07:28:20.628763 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:20.628519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:58821)
0810 07:28:20.628666 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:20.628715 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260810 07:28:21.420233 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:21.419941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:54245)
0810 07:28:21.420138 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:21.420185 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260810 07:28:21.483654 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:21.483418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:50459)
0810 07:28:21.483563 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:21.483604 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260810 07:28:21.629623 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:21.629380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:50831)
0810 07:28:21.629523 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:21.629571 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260810 07:28:22.421134 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:22.420858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:36343)
0810 07:28:22.421032 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:22.421077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260810 07:28:22.484474 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:22.484209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:41095)
0810 07:28:22.484381 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:22.484426 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260810 07:28:22.630573 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:22.630311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:52761)
0810 07:28:22.630466 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:22.630513 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260810 07:28:23.421976 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:23.421724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:34981)
0810 07:28:23.421895 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:23.421929 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260810 07:28:23.485368 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:23.485121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:56951)
0810 07:28:23.485282 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:23.485316 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260810 07:28:23.631481 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:23.631213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:56593)
0810 07:28:23.631374 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:23.631421 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260810 07:28:24.422694 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:24.422448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:48519)
0810 07:28:24.422609 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:24.422646 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260810 07:28:24.486122 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:24.485880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:56319)
0810 07:28:24.486042 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:24.486077 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260810 07:28:24.632308 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:24.632032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:43217)
0810 07:28:24.632207 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:24.632256 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260810 07:28:25.423635 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:25.423329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:51749)
0810 07:28:25.423543 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:25.423585 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":128,"threads_started":1}
I20260810 07:28:25.487043 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:25.486646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:60645)
0810 07:28:25.486913 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:25.486975 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260810 07:28:25.633258 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:25.632985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:42765)
0810 07:28:25.633141 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:25.633206 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260810 07:28:26.424588 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:26.424339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:34657)
0810 07:28:26.424500 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:26.424539 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260810 07:28:26.487975 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:26.487680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:55049)
0810 07:28:26.487853 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:26.487912 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260810 07:28:26.634131 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:26.633834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:52839)
0810 07:28:26.634032 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:26.634080 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260810 07:28:27.425483 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:27.425238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:32935)
0810 07:28:27.425386 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:27.425436 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260810 07:28:27.488785 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:27.488531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:34253)
0810 07:28:27.488682 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:27.488734 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260810 07:28:27.634963 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:27.634704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:39931)
0810 07:28:27.634865 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:27.634907 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260810 07:28:28.426288 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:28.426060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:53401)
0810 07:28:28.426199 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:28.426240 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260810 07:28:28.489584 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:28.489348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:56191)
0810 07:28:28.489497 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:28.489536 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260810 07:28:28.635699 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:28.635471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:53683)
0810 07:28:28.635610 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:28.635649 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260810 07:28:29.427204 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:29.426965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.193:46389)
0810 07:28:29.427118 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:29.427156 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260810 07:28:29.490536 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:29.490251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.194:49941)
0810 07:28:29.490438 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:29.490487 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260810 07:28:29.634382 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 11681
I20260810 07:28:29.638224 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:29.637957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:36423 (local address 127.24.60.195:42599)
0810 07:28:29.638120 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:29.638171 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.254:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260810 07:28:29.642283 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 11869
I20260810 07:28:29.649920 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12025
2026-08-10T07:28:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58917 ms)
[----------] 30 tests from SecurityITest (246391 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:29 dist-test-slave-vxj2 krb5kdc[12229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:29 dist-test-slave-vxj2 krb5kdc[12229](info): set up 2 sockets
Aug 10 07:28:29 dist-test-slave-vxj2 krb5kdc[12229](info): commencing operation
krb5kdc: starting...
W20260810 07:28:30.777227 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:30 dist-test-slave-vxj2 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:28:30.908942 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:43219
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:30.993806 12244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:30.994084 12244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:30.994124 12244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:30.995308 12244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:30.995389 12244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:30.995415 12244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:30.995436 12244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:30.995466 12244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:30.997000 12244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:30.997323 12244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:30.997584 12244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.003258 12250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.003273 12252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.003268 12249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.003307 12244 server_base.cc:1034] running on GCE node
I20260810 07:28:31.003756 12244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.004077 12244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.005232 12244 hybrid_clock.cc:648] HybridClock initialized: now 1786346911005194 us; error 47 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.007898 12244 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:28:31.008428 12244 webserver.cc:460] Webserver started at http://127.24.60.254:46143/ using document root <none> and password file <none>
I20260810 07:28:31.008692 12244 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.008766 12244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.008921 12244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:31.010128 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "25754c85c23746329015445763b34ee7"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "fe83cc4bf56754de1f76e3b03b61650e"
server_key_iv: "4c074f476f721bd902d407603c772bea"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.010516 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "25754c85c23746329015445763b34ee7"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "fe83cc4bf56754de1f76e3b03b61650e"
server_key_iv: "4c074f476f721bd902d407603c772bea"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.012005 12244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:28:31.012816 12260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.013083 12244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:31.013229 12244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "25754c85c23746329015445763b34ee7"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "fe83cc4bf56754de1f76e3b03b61650e"
server_key_iv: "4c074f476f721bd902d407603c772bea"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.013316 12244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.027798 12244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.030046 12244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.030237 12244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.034833 12244 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:43219
I20260810 07:28:31.034890 12324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:43219 every 8 connection(s)
I20260810 07:28:31.035272 12244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:31.036235 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.038571 12325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7: Bootstrap starting.
I20260810 07:28:31.039119 12325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.039367 12325 log.cc:824] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.040099 12325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7: No bootstrap required, opened a new log
I20260810 07:28:31.041800 12325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25754c85c23746329015445763b34ee7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43219 } }
I20260810 07:28:31.041944 12325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.041967 12325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25754c85c23746329015445763b34ee7, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.042110 12325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25754c85c23746329015445763b34ee7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43219 } }
I20260810 07:28:31.042193 12325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:31.042218 12325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:31.042248 12325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:31.042753 12325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25754c85c23746329015445763b34ee7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43219 } }
I20260810 07:28:31.042845 12325 leader_election.cc:302] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25754c85c23746329015445763b34ee7; no voters:
I20260810 07:28:31.042979 12325 leader_election.cc:288] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:31.043035 12328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:31.043226 12328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 25754c85c23746329015445763b34ee7, State: Running, Role: LEADER
I20260810 07:28:31.043267 12325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:31.043385 12328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25754c85c23746329015445763b34ee7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43219 } }
I20260810 07:28:31.044011 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25754c85c23746329015445763b34ee7. Latest consensus state: current_term: 1 leader_uuid: "25754c85c23746329015445763b34ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25754c85c23746329015445763b34ee7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43219 } } }
I20260810 07:28:31.044116 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:31.044349 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25754c85c23746329015445763b34ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25754c85c23746329015445763b34ee7" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43219 } } }
I20260810 07:28:31.044395 12339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:31.044494 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:31.044960 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12244
I20260810 07:28:31.045042 12339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:31.045079 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:31.045351 24819 external_mini_cluster.cc:1428] Setting key d4a9e661df4d7ef4355cc99a114b4f24
I20260810 07:28:31.047004 12339 catalog_manager.cc:1347] Generated new cluster ID: d11c1877d88e48d5b6601aed4d8489fc
I20260810 07:28:31.047053 12339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:31.057072 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.045872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.24.60.254:43219)
0810 07:28:31.046019 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.046027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.046474 (+ 447us) server_negotiation.cc:407] Connection header received
0810 07:28:31.046649 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.046656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.046714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.047686 (+ 972us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.047786 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.049243 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.050396 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.050796 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.050919 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.053521 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.053526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.053526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.053540 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.055379 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.055769 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.055772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.055773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.055873 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.056097 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.056098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.056098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.056391 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.056565 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.056666 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.056825 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260810 07:28:31.062170 12339 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:31.062945 12339 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:31.072853 12339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7: Generated new TSK 0
I20260810 07:28:31.073081 12339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:28:31.108479 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43219
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:31.213764 12353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.214005 12353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.214031 12353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.215348 12353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.215404 12353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.215433 12353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:31.217319 12353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:31.217600 12353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.217872 12353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.223408 12359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.223399 12361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.223630 12353 server_base.cc:1034] running on GCE node
W20260810 07:28:31.223397 12358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.223973 12353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.224260 12353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.225427 12353 hybrid_clock.cc:648] HybridClock initialized: now 1786346911225395 us; error 55 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.228188 12353 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:28:31.228621 12353 webserver.cc:460] Webserver started at http://127.24.60.193:39243/ using document root <none> and password file <none>
I20260810 07:28:31.228816 12353 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.228856 12353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.228952 12353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:31.230085 12353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "2f8008c89b444acfa9612618c04973c6"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "502798a4597848a886aa0d030ff4b294"
server_key_iv: "d268b2d249578c88ee38ef9db13812d1"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.230396 12353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "2f8008c89b444acfa9612618c04973c6"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "502798a4597848a886aa0d030ff4b294"
server_key_iv: "d268b2d249578c88ee38ef9db13812d1"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.231750 12353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:31.232511 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.232827 12353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:31.232929 12353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "2f8008c89b444acfa9612618c04973c6"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "502798a4597848a886aa0d030ff4b294"
server_key_iv: "d268b2d249578c88ee38ef9db13812d1"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.232993 12353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.257959 12353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.260170 12353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.260298 12353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.260520 12353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:31.260893 12353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:31.260931 12353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.260953 12353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:31.261003 12353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.267252 12353 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46555
I20260810 07:28:31.267349 12507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46555 every 8 connection(s)
I20260810 07:28:31.267741 12353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:31.275516 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12353
I20260810 07:28:31.275655 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:31.275900 24819 external_mini_cluster.cc:1428] Setting key 7a0db28e73526282ac80272925de98be
I20260810 07:28:31.278856 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.268635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:44983 (local address 127.24.60.254:43219)
0810 07:28:31.268690 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.268692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.268956 (+ 264us) server_negotiation.cc:407] Connection header received
0810 07:28:31.269533 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.269534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.269551 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.269620 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.270640 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.271166 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.272362 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.272582 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.272647 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.275226 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.275228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.275228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.275268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.277048 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.277667 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.277668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.277669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.277727 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.278033 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.278034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.278034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.278144 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.278193 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.278608 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.278726 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:28:31.278956 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.268591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:43219 (local address 127.24.60.193:44983)
0810 07:28:31.268780 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.268840 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.269397 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.269679 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.269682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.270310 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:31.270531 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.270536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.271284 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.271285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.272242 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.272245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.272687 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.272688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.272817 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.273533 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:31.273552 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:31.275118 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:31.277141 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.277178 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.277181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.277580 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.277793 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.277794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.277795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.277961 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.278207 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:31.278214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:31.278407 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.278539 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.278749 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260810 07:28:31.279547 12508 heartbeater.cc:347] Connected to a master server at 127.24.60.254:43219
I20260810 07:28:31.279675 12508 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:31.279884 12508 heartbeater.cc:511] Master 127.24.60.254:43219 requested a full tablet report, sending...
I20260810 07:28:31.280517 12280 ts_manager.cc:194] Registered new tserver with Master: 2f8008c89b444acfa9612618c04973c6 (127.24.60.193:46555)
I20260810 07:28:31.281366 12280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:44983
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:28:31.305050 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43219
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:31.395272 12513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.395581 12513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.395632 12513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.397006 12513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.397095 12513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.397146 12513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:31.398909 12513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:31.399276 12513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.399588 12513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.405256 12519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.405259 12518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.405259 12521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.405558 12513 server_base.cc:1034] running on GCE node
I20260810 07:28:31.405898 12513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.406136 12513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.407305 12513 hybrid_clock.cc:648] HybridClock initialized: now 1786346911407293 us; error 33 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.410238 12513 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:28:31.410748 12513 webserver.cc:460] Webserver started at http://127.24.60.194:43613/ using document root <none> and password file <none>
I20260810 07:28:31.410964 12513 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.411006 12513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.411101 12513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:31.412146 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "01b7e9280725492785722938160c776a"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "64d95a8ff71f970998516d86acd9c4f6"
server_key_iv: "79f25be67933750a685dd12dd14c176d"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.412462 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "01b7e9280725492785722938160c776a"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "64d95a8ff71f970998516d86acd9c4f6"
server_key_iv: "79f25be67933750a685dd12dd14c176d"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.414067 12513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:31.415074 12530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.415313 12513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:31.415426 12513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "01b7e9280725492785722938160c776a"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "64d95a8ff71f970998516d86acd9c4f6"
server_key_iv: "79f25be67933750a685dd12dd14c176d"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.415501 12513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.443331 12513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.446120 12513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.446298 12513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.446586 12513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:31.446949 12513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:31.446985 12513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.447037 12513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:31.447067 12513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.453101 12513 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:43943
I20260810 07:28:31.453195 12664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:43943 every 8 connection(s)
I20260810 07:28:31.453569 12513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:31.461344 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12513
I20260810 07:28:31.461496 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:31.461704 24819 external_mini_cluster.cc:1428] Setting key 4ef370a5dd35bd23b27b47ac86f3eedc
I20260810 07:28:31.464246 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.454554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:36137 (local address 127.24.60.254:43219)
0810 07:28:31.454649 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.454651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.454836 (+ 185us) server_negotiation.cc:407] Connection header received
0810 07:28:31.455372 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.455373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.455384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.455446 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.456446 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.456979 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.458154 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.458357 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.458420 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.460805 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.460806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.460807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.460815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.462672 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.463162 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.463163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.463164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.463236 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.463508 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.463509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.463510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.463664 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.463728 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.464057 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.464151 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:28:31.464479 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.454504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:43219 (local address 127.24.60.194:36137)
0810 07:28:31.454695 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.454735 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.455251 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.455513 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.455516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.456125 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:31.456357 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.456363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.457092 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.457093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.458057 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.458060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.458444 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.458445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.458571 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.459202 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:31.459211 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:31.460705 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:31.462767 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.462777 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.462779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.463073 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.463314 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.463315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.463316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.463437 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.463731 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:31.463759 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:31.463904 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.464021 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.464210 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260810 07:28:31.465131 12665 heartbeater.cc:347] Connected to a master server at 127.24.60.254:43219
I20260810 07:28:31.465294 12665 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:31.465523 12665 heartbeater.cc:511] Master 127.24.60.254:43219 requested a full tablet report, sending...
I20260810 07:28:31.465966 12280 ts_manager.cc:194] Registered new tserver with Master: 01b7e9280725492785722938160c776a (127.24.60.194:43943)
I20260810 07:28:31.466498 12280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:36137
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:28:31.490149 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43219
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:31.586747 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.587136 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.587174 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.588737 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.588822 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.588864 12670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:31.590682 12670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:31.590981 12670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.591248 12670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.597198 12676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.597222 12679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.597211 12675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.597455 12670 server_base.cc:1034] running on GCE node
I20260810 07:28:31.597676 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.597994 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.599152 12670 hybrid_clock.cc:648] HybridClock initialized: now 1786346911599134 us; error 53 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.602017 12670 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:28:31.602510 12670 webserver.cc:460] Webserver started at http://127.24.60.195:41385/ using document root <none> and password file <none>
I20260810 07:28:31.602826 12670 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.602908 12670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.603060 12670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:31.604578 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "6cc333a3915e4f34918b1706fef3c97e"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "69bd0a92552ca65e638d2591ff44ebc2"
server_key_iv: "b2b37adbc13d02dd0bed1e0a15b2736a"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.605037 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "6cc333a3915e4f34918b1706fef3c97e"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "69bd0a92552ca65e638d2591ff44ebc2"
server_key_iv: "b2b37adbc13d02dd0bed1e0a15b2736a"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.607147 12670 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 07:28:31.608103 12687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.608278 12670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:31.608394 12670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "6cc333a3915e4f34918b1706fef3c97e"
format_stamp: "Formatted at 2026-08-10 07:28:31 on dist-test-slave-vxj2"
server_key: "69bd0a92552ca65e638d2591ff44ebc2"
server_key_iv: "b2b37adbc13d02dd0bed1e0a15b2736a"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.608491 12670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.621439 12670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.623667 12670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.623831 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.624115 12670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:31.624464 12670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:31.624500 12670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.624523 12670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:31.624557 12670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.630743 12670 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:41901
I20260810 07:28:31.630822 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:41901 every 8 connection(s)
I20260810 07:28:31.631160 12670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:31.636317 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12670
I20260810 07:28:31.636431 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:31.636659 24819 external_mini_cluster.cc:1428] Setting key 439720b87f068c7449a70fbbd56ec1e8
Aug 10 07:28:31 dist-test-slave-vxj2 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:31.641255 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.631985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:55377 (local address 127.24.60.254:43219)
0810 07:28:31.632056 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.632058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.632312 (+ 254us) server_negotiation.cc:407] Connection header received
0810 07:28:31.632924 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.632926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.632939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.633000 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.634101 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.634614 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.635647 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.635851 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.635940 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.638599 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.638600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.638601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.638609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.639784 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.640199 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.640200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.640201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.640266 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.640535 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.640536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.640536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.640659 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.640738 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.641024 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.641110 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:31.641429 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.631988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:43219 (local address 127.24.60.195:55377)
0810 07:28:31.632187 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.632224 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.632797 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.633079 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.633083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.633761 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:31.633969 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.633975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.634718 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.634719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.635547 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.635549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.635952 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.635953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.636084 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.636815 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:31.636831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:31.638501 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:31.639868 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.639876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.639878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.640118 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.640342 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.640343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.640343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.640454 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.640727 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:31.640735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:31.640880 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.640999 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.641160 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260810 07:28:31.641911 12825 heartbeater.cc:347] Connected to a master server at 127.24.60.254:43219
I20260810 07:28:31.642036 12825 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:31.642220 12825 heartbeater.cc:511] Master 127.24.60.254:43219 requested a full tablet report, sending...
I20260810 07:28:31.642597 12280 ts_manager.cc:194] Registered new tserver with Master: 6cc333a3915e4f34918b1706fef3c97e (127.24.60.195:41901)
I20260810 07:28:31.642980 12280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:55377
I20260810 07:28:31.648523 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:31.656162 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.649899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.24.60.254:43219)
0810 07:28:31.649981 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.649983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.650066 (+ 83us) server_negotiation.cc:407] Connection header received
0810 07:28:31.650157 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.650158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.650168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.650225 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.650786 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.651288 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.652029 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.652241 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.652326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.653562 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.653564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.653564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.653572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.654883 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.655226 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.655227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.655227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.655315 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.655540 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.655541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.655541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.655700 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.655802 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.655945 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.656055 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:28:31.657848 12280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:31.659605 12280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:31.666481 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.662187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43943 (local address 127.0.0.1:38068)
0810 07:28:31.662304 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.662346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.662413 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.663073 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.663074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.663077 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.663135 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.663137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.664073 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.664074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.665387 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.665390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.666135 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.666136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.666216 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.666230 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.666333 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.666392 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":27,"thread_start_us":67,"threads_started":1}
I20260810 07:28:31.666493 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.662613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.24.60.194:43943)
0810 07:28:31.662879 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.662885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.662896 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:28:31.662940 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.662943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.662968 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.663231 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.663293 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.663954 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.665499 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.666044 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.666136 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.666202 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.666352 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.666394 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260810 07:28:31.666754 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.662414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46555 (local address 127.0.0.1:45346)
0810 07:28:31.662544 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.662554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.662601 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.663071 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.663072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.663074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.663110 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.663112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.664442 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.664443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.665527 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.665530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.666452 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.666454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.666545 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.666552 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.666586 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.666638 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":26,"thread_start_us":97,"threads_started":1}
I20260810 07:28:31.666977 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.662253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41901 (local address 127.0.0.1:45144)
0810 07:28:31.662425 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.662436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.662508 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.662848 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.662853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.662856 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.662943 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.662945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.664393 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.664394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.665513 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.665515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.666454 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.666455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.666545 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.666554 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.666731 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.666767 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260810 07:28:31.667271 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.662380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.24.60.195:41901)
0810 07:28:31.662635 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.662640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.662650 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:31.662699 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.662702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.662721 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.663033 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.663115 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.664267 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.665603 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.666337 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.666921 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.666997 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.667109 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.667180 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
I20260810 07:28:31.667423 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.662678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.24.60.193:46555)
0810 07:28:31.662886 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.662891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.662900 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:28:31.662945 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.662947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.662963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.663185 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.663255 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.664360 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.665643 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.666348 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.667162 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.667230 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.667257 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.667310 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260810 07:28:31.667279 12583 tablet_service.cc:1467] Processing CreateTablet for tablet 06776ae838034d72a431444a50264bea (DEFAULT_TABLE table=test-table [id=730fb77ef3f140a294dae5d19d781fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:31.667713 12583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06776ae838034d72a431444a50264bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.668046 12740 tablet_service.cc:1467] Processing CreateTablet for tablet 06776ae838034d72a431444a50264bea (DEFAULT_TABLE table=test-table [id=730fb77ef3f140a294dae5d19d781fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:31.668366 12740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06776ae838034d72a431444a50264bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.669731 12427 tablet_service.cc:1467] Processing CreateTablet for tablet 06776ae838034d72a431444a50264bea (DEFAULT_TABLE table=test-table [id=730fb77ef3f140a294dae5d19d781fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:31.670143 12427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06776ae838034d72a431444a50264bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.671442 12844 tablet_bootstrap.cc:492] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Bootstrap starting.
I20260810 07:28:31.671763 12845 tablet_bootstrap.cc:492] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Bootstrap starting.
I20260810 07:28:31.672303 12844 tablet_bootstrap.cc:654] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.672617 12845 tablet_bootstrap.cc:654] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.672927 12844 log.cc:824] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.673198 12845 log.cc:824] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.673498 12847 tablet_bootstrap.cc:492] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: Bootstrap starting.
I20260810 07:28:31.673822 12844 tablet_bootstrap.cc:492] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: No bootstrap required, opened a new log
I20260810 07:28:31.673914 12844 ts_tablet_manager.cc:1397] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 07:28:31.674130 12847 tablet_bootstrap.cc:654] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.674623 12847 log.cc:824] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.675482 12847 tablet_bootstrap.cc:492] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: No bootstrap required, opened a new log
I20260810 07:28:31.675555 12847 ts_tablet_manager.cc:1397] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:31.675750 12844 raft_consensus.cc:348] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.675930 12844 raft_consensus.cc:374] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.675963 12844 raft_consensus.cc:729] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b7e9280725492785722938160c776a, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.676084 12844 consensus_queue.cc:260] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.676363 12844 ts_tablet_manager.cc:1428] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 07:28:31.676424 12665 heartbeater.cc:503] Master 127.24.60.254:43219 was elected leader, sending a full tablet report...
I20260810 07:28:31.677637 12845 tablet_bootstrap.cc:492] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: No bootstrap required, opened a new log
I20260810 07:28:31.677558 12847 raft_consensus.cc:348] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.677733 12847 raft_consensus.cc:374] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.677745 12845 ts_tablet_manager.cc:1397] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 07:28:31.677778 12847 raft_consensus.cc:729] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8008c89b444acfa9612618c04973c6, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.677913 12847 consensus_queue.cc:260] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.678193 12847 ts_tablet_manager.cc:1428] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:31.678269 12508 heartbeater.cc:503] Master 127.24.60.254:43219 was elected leader, sending a full tablet report...
I20260810 07:28:31.679585 12845 raft_consensus.cc:348] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.679775 12845 raft_consensus.cc:374] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.679831 12845 raft_consensus.cc:729] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc333a3915e4f34918b1706fef3c97e, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.680006 12845 consensus_queue.cc:260] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.680363 12845 ts_tablet_manager.cc:1428] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:31.680368 12825 heartbeater.cc:503] Master 127.24.60.254:43219 was elected leader, sending a full tablet report...
W20260810 07:28:31.704262 12666 tablet.cc:2369] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:31.768618 12509 tablet.cc:2369] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:31.881778 12826 tablet.cc:2369] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:31.976184 12850 raft_consensus.cc:482] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:31.976322 12850 raft_consensus.cc:504] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.976811 12850 leader_election.cc:288] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc333a3915e4f34918b1706fef3c97e (127.24.60.195:41901), 2f8008c89b444acfa9612618c04973c6 (127.24.60.193:46555)
I20260810 07:28:31.980476 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.977277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:43315 (local address 127.24.60.193:46555)
0810 07:28:31.977327 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.977329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.977339 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:31.977373 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.977374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.977387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.977458 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.977657 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.978441 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.979736 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.980109 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.980198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.980208 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.980363 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.980403 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:28:31.980499 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.977063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46555 (local address 127.24.60.194:43315)
0810 07:28:31.977252 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.977261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.977307 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.977479 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.977480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.977482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.977565 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.977567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.978529 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.978530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.979639 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.979641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.980197 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.980198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.980282 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.980286 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.980354 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.980400 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":15,"thread_start_us":68,"threads_started":1}
I20260810 07:28:31.980784 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.977117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:43793 (local address 127.24.60.195:41901)
0810 07:28:31.977195 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.977197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.977272 (+ 75us) server_negotiation.cc:407] Connection header received
0810 07:28:31.977393 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.977400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.977408 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.977455 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.977680 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.978187 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.979647 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.980333 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.980412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.980422 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.980646 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.980693 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:28:31.980960 12451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06776ae838034d72a431444a50264bea" candidate_uuid: "01b7e9280725492785722938160c776a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8008c89b444acfa9612618c04973c6" is_pre_election: true
I20260810 07:28:31.981086 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.977004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:41901 (local address 127.24.60.194:43793)
0810 07:28:31.977195 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.977207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.977306 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.977531 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.977532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.977534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.977592 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.977594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.978289 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.978290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.979530 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.979534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.980462 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.980463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.980555 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.980560 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.980916 (+ 356us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.980982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260810 07:28:31.981101 12451 raft_consensus.cc:2454] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b7e9280725492785722938160c776a in term 0.
I20260810 07:28:31.981428 12534 leader_election.cc:302] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b7e9280725492785722938160c776a, 2f8008c89b444acfa9612618c04973c6; no voters:
I20260810 07:28:31.981503 12762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06776ae838034d72a431444a50264bea" candidate_uuid: "01b7e9280725492785722938160c776a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc333a3915e4f34918b1706fef3c97e" is_pre_election: true
I20260810 07:28:31.981611 12850 raft_consensus.cc:2781] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:31.981654 12762 raft_consensus.cc:2454] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b7e9280725492785722938160c776a in term 0.
I20260810 07:28:31.981689 12850 raft_consensus.cc:482] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:31.981714 12850 raft_consensus.cc:3037] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:31.982492 12850 raft_consensus.cc:504] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.982651 12850 leader_election.cc:288] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [CANDIDATE]: Term 1 election: Requested vote from peers 6cc333a3915e4f34918b1706fef3c97e (127.24.60.195:41901), 2f8008c89b444acfa9612618c04973c6 (127.24.60.193:46555)
I20260810 07:28:31.982800 12762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06776ae838034d72a431444a50264bea" candidate_uuid: "01b7e9280725492785722938160c776a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc333a3915e4f34918b1706fef3c97e"
I20260810 07:28:31.982918 12762 raft_consensus.cc:3037] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:31.982945 12451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06776ae838034d72a431444a50264bea" candidate_uuid: "01b7e9280725492785722938160c776a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8008c89b444acfa9612618c04973c6"
I20260810 07:28:31.983023 12451 raft_consensus.cc:3037] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:31.983916 12762 raft_consensus.cc:2454] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b7e9280725492785722938160c776a in term 1.
I20260810 07:28:31.983916 12451 raft_consensus.cc:2454] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b7e9280725492785722938160c776a in term 1.
I20260810 07:28:31.984117 12534 leader_election.cc:302] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b7e9280725492785722938160c776a, 2f8008c89b444acfa9612618c04973c6; no voters:
I20260810 07:28:31.984215 12850 raft_consensus.cc:2781] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:31.984407 12850 raft_consensus.cc:686] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 1 LEADER]: Becoming Leader. State: Replica: 01b7e9280725492785722938160c776a, State: Running, Role: LEADER
I20260810 07:28:31.984520 12850 consensus_queue.cc:237] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } }
I20260810 07:28:31.985477 12280 catalog_manager.cc:5495] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a reported cstate change: term changed from 0 to 1, leader changed from <none> to 01b7e9280725492785722938160c776a (127.24.60.194). New cstate: current_term: 1 leader_uuid: "01b7e9280725492785722938160c776a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b7e9280725492785722938160c776a" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 } health_report { overall_health: UNKNOWN } } }
I20260810 07:28:32.022925 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.020602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.24.60.194:43943)
0810 07:28:32.020675 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.020677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.020707 (+ 30us) server_negotiation.cc:407] Connection header received
0810 07:28:32.020752 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.020753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.020773 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.020828 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:32.020950 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.021513 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.022253 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.022437 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.022499 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.022647 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:32.022752 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:32.022812 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.022844 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.022862 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:28:32.024921 12451 raft_consensus.cc:1261] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 1 FOLLOWER]: Refusing update from remote peer 01b7e9280725492785722938160c776a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:32.024930 12762 raft_consensus.cc:1261] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 1 FOLLOWER]: Refusing update from remote peer 01b7e9280725492785722938160c776a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:32.025314 12850 consensus_queue.cc:1035] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8008c89b444acfa9612618c04973c6" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:32.025511 12857 consensus_queue.cc:1035] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc333a3915e4f34918b1706fef3c97e" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:32.030565 12279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:32.030687 12279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:32.031522 12279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25754c85c23746329015445763b34ee7: Sending DeleteTablet for 3 replicas of tablet 06776ae838034d72a431444a50264bea
I20260810 07:28:32.031911 12740 tablet_service.cc:1514] Processing DeleteTablet for tablet 06776ae838034d72a431444a50264bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:32 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:45144
I20260810 07:28:32.031955 12427 tablet_service.cc:1514] Processing DeleteTablet for tablet 06776ae838034d72a431444a50264bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:32 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:45346
I20260810 07:28:32.032217 12870 tablet_replica.cc:330] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6: stopping tablet replica
I20260810 07:28:32.032325 12870 raft_consensus.cc:2229] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:32.032398 12583 tablet_service.cc:1514] Processing DeleteTablet for tablet 06776ae838034d72a431444a50264bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:32 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:38068
I20260810 07:28:32.032536 12870 raft_consensus.cc:2258] T 06776ae838034d72a431444a50264bea P 2f8008c89b444acfa9612618c04973c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:32.032603 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12353
I20260810 07:28:32.034121 12872 tablet_replica.cc:330] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: stopping tablet replica
I20260810 07:28:32.034291 12872 raft_consensus.cc:2229] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:28:32.034495 12872 raft_consensus.cc:2258] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:32.041759 12872 ts_tablet_manager.cc:1905] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:32.042433 12871 tablet_replica.cc:330] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: stopping tablet replica
I20260810 07:28:32.042590 12871 raft_consensus.cc:2229] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:32.042724 12871 raft_consensus.cc:2258] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 07:28:32.044039 12265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:46555 (error 108)
I20260810 07:28:32.044085 12872 ts_tablet_manager.cc:1918] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:32.044181 12872 log.cc:1196] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/06776ae838034d72a431444a50264bea
I20260810 07:28:32.044523 12872 ts_tablet_manager.cc:1939] T 06776ae838034d72a431444a50264bea P 01b7e9280725492785722938160c776a: Deleting consensus metadata
I20260810 07:28:32.044718 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12513
I20260810 07:28:32.049922 12871 ts_tablet_manager.cc:1905] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:32.050838 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.050422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46555 (local address 127.0.0.1:45362)
0810 07:28:32.050594 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.050769 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260810 07:28:32.051177 12265 catalog_manager.cc:4584] TS 2f8008c89b444acfa9612618c04973c6 (127.24.60.193:46555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06776ae838034d72a431444a50264bea: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46555: connect: Connection refused (error 111)
I20260810 07:28:32.051921 12871 ts_tablet_manager.cc:1918] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:32.052011 12871 log.cc:1196] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/06776ae838034d72a431444a50264bea
I20260810 07:28:32.052321 12871 ts_tablet_manager.cc:1939] T 06776ae838034d72a431444a50264bea P 6cc333a3915e4f34918b1706fef3c97e: Deleting consensus metadata
I20260810 07:28:32.052945 12264 catalog_manager.cc:4841] TS 6cc333a3915e4f34918b1706fef3c97e (127.24.60.195:41901): tablet 06776ae838034d72a431444a50264bea (table test-table [id=730fb77ef3f140a294dae5d19d781fde]) successfully deleted
I20260810 07:28:32.054739 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12670
I20260810 07:28:32.054785 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.054654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43943 (local address 127.0.0.1:38078)
0810 07:28:32.054714 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.054743 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.194:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 07:28:32.063294 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12244
2026-08-10T07:28:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2412 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T07:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:32Z Disabled control of system clock
I20260810 07:28:32.094833 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:43099
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:43099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:32.177754 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:32.178004 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:32.178026 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:32.179171 12879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:32.179209 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:32.179222 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:32.179235 12879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:32.179247 12879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:32.180573 12879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:43099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:43099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:32.180786 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:32.181038 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:32.186213 12887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:32.186201 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:32.186178 12889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:32.186669 12879 server_base.cc:1034] running on GCE node
I20260810 07:28:32.186967 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:32.187284 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:32.188438 12879 hybrid_clock.cc:648] HybridClock initialized: now 1786346912188414 us; error 41 us; skew 500 ppm
I20260810 07:28:32.189886 12879 webserver.cc:460] Webserver started at http://127.24.60.254:43413/ using document root <none> and password file <none>
I20260810 07:28:32.190186 12879 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:32.190237 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:32.190431 12879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:32.191588 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "8883b92aaeab42929c9b78736ab04b53"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "377ffcbafab7d3e713104426e97fdd38"
server_key_iv: "c7799154804be824ca532e94212a8d26"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.191983 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "8883b92aaeab42929c9b78736ab04b53"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "377ffcbafab7d3e713104426e97fdd38"
server_key_iv: "c7799154804be824ca532e94212a8d26"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.193526 12879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:28:32.194347 12895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.194537 12879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:28:32.194653 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "8883b92aaeab42929c9b78736ab04b53"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "377ffcbafab7d3e713104426e97fdd38"
server_key_iv: "c7799154804be824ca532e94212a8d26"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.194732 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:32.210995 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:32.211385 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:32.211570 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:32.215952 12879 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:43099
I20260810 07:28:32.215994 12953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:43099 every 8 connection(s)
I20260810 07:28:32.216396 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:32.217550 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:32.219571 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12879
I20260810 07:28:32.219698 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:32.219898 24819 external_mini_cluster.cc:1428] Setting key 1d55d690d09df9cd393a6e0cc355f712
I20260810 07:28:32.220392 12954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53: Bootstrap starting.
I20260810 07:28:32.221133 12954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:32.221565 12954 log.cc:824] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:32.222369 12954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53: No bootstrap required, opened a new log
I20260810 07:28:32.224769 12954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8883b92aaeab42929c9b78736ab04b53" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43099 } }
I20260810 07:28:32.224936 12954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:32.224962 12954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883b92aaeab42929c9b78736ab04b53, State: Initialized, Role: FOLLOWER
I20260810 07:28:32.225142 12954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8883b92aaeab42929c9b78736ab04b53" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43099 } }
I20260810 07:28:32.225270 12954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:32.225327 12954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:32.225378 12954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:32.226099 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.220511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.24.60.254:43099)
0810 07:28:32.220754 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.220762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.220789 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:28:32.221317 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.221327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.221411 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.221545 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:32.221804 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.222831 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.223973 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.224398 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.224499 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.225315 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:32.225319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:32.225320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:32.225559 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:32.225595 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:32.225607 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:32.225661 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.225781 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.225847 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260810 07:28:32.226172 12954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8883b92aaeab42929c9b78736ab04b53" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43099 } }
I20260810 07:28:32.226317 12954 leader_election.cc:302] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8883b92aaeab42929c9b78736ab04b53; no voters:
I20260810 07:28:32.226536 12954 leader_election.cc:288] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:32.226578 12961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:32.226742 12961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [term 1 LEADER]: Becoming Leader. State: Replica: 8883b92aaeab42929c9b78736ab04b53, State: Running, Role: LEADER
I20260810 07:28:32.226814 12954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:32.226855 12961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8883b92aaeab42929c9b78736ab04b53" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43099 } }
I20260810 07:28:32.227191 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8883b92aaeab42929c9b78736ab04b53. Latest consensus state: current_term: 1 leader_uuid: "8883b92aaeab42929c9b78736ab04b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8883b92aaeab42929c9b78736ab04b53" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43099 } } }
I20260810 07:28:32.227291 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:32.227308 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8883b92aaeab42929c9b78736ab04b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8883b92aaeab42929c9b78736ab04b53" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 43099 } } }
I20260810 07:28:32.227423 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:32.227864 12972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:32.228449 12972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:32.230049 12972 catalog_manager.cc:1347] Generated new cluster ID: 61b0434046ab476ab021d50960a29caa
I20260810 07:28:32.230108 12972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:32.257031 12972 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:32.257849 12972 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:32.262534 12972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53: Generated new TSK 0
I20260810 07:28:32.262742 12972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:32.264817 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43099
--builtin_ntp_servers=127.24.60.212:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:32.351131 12985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:32.351405 12985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:32.351449 12985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:32.352703 12985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:32.352784 12985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:32.352828 12985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:32.354368 12985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:32.354681 12985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:32.355026 12985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:32.360422 12993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:32.360419 12990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:32.360417 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:32.360630 12985 server_base.cc:1034] running on GCE node
I20260810 07:28:32.360926 12985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:32.361267 12985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:32.362442 12985 hybrid_clock.cc:648] HybridClock initialized: now 1786346912362417 us; error 42 us; skew 500 ppm
I20260810 07:28:32.363770 12985 webserver.cc:460] Webserver started at http://127.24.60.193:44441/ using document root <none> and password file <none>
I20260810 07:28:32.364040 12985 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:32.364089 12985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:32.364184 12985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:32.365336 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "15c529d166594994bbaff974fb49b7e3"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "fffea8314699b80908a856d7eaadef24"
server_key_iv: "6ce3580406af3832eff4dd12b0c87705"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.365661 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "15c529d166594994bbaff974fb49b7e3"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "fffea8314699b80908a856d7eaadef24"
server_key_iv: "6ce3580406af3832eff4dd12b0c87705"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.367141 12985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:28:32.367898 12999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.368090 12985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:28:32.368186 12985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "15c529d166594994bbaff974fb49b7e3"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "fffea8314699b80908a856d7eaadef24"
server_key_iv: "6ce3580406af3832eff4dd12b0c87705"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.368247 12985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:32.382445 12985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:32.382746 12985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:32.382851 12985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:32.383059 12985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:32.383424 12985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:32.383463 12985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.383526 12985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:32.383553 12985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.389976 12985 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:46843
I20260810 07:28:32.390064 13129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:46843 every 8 connection(s)
I20260810 07:28:32.390471 12985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:32.391112 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 12985
I20260810 07:28:32.391209 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:32.391372 24819 external_mini_cluster.cc:1428] Setting key d5d4821b6cb3922322827cfdc087c50e
I20260810 07:28:32.392553 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43099
--builtin_ntp_servers=127.24.60.212:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:28:32.397799 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.392031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:51773 (local address 127.24.60.254:43099)
0810 07:28:32.392091 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.392093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.392374 (+ 281us) server_negotiation.cc:407] Connection header received
0810 07:28:32.392956 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.392957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.392970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.393047 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:32.393834 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.394494 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.396102 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.396294 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.396349 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.397441 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:32.397443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:32.397443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:32.397449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:32.397478 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:32.397482 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:32.397530 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.397710 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.397736 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:28:32.398141 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.392006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:43099 (local address 127.24.60.193:51773)
0810 07:28:32.392226 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.392271 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:32.392841 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:32.393109 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:32.393112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:32.393121 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:32.393732 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.393738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.394620 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.394622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.395997 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.396002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.396405 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.396406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.396619 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.397251 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:32.397252 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:32.397299 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:32.397340 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:32.397577 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:32.397585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:32.397589 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:32.397710 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 07:28:32.397845 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260810 07:28:32.398838 13130 heartbeater.cc:347] Connected to a master server at 127.24.60.254:43099
I20260810 07:28:32.398968 13130 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:32.399219 13130 heartbeater.cc:511] Master 127.24.60.254:43099 requested a full tablet report, sending...
I20260810 07:28:32.399834 12915 ts_manager.cc:194] Registered new tserver with Master: 15c529d166594994bbaff974fb49b7e3 (127.24.60.193:46843)
I20260810 07:28:32.400493 12915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.193:51773
W20260810 07:28:32.481220 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:32.481456 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:32.481482 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:32.482728 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:32.482789 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:32.482818 13134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:32.484221 13134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:32.484508 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:32.484853 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:32.490453 13143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:32.490460 13141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:32.490499 13134 server_base.cc:1034] running on GCE node
W20260810 07:28:32.490504 13140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:32.490973 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:32.491242 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:32.492455 13134 hybrid_clock.cc:648] HybridClock initialized: now 1786346912492420 us; error 67 us; skew 500 ppm
I20260810 07:28:32.493906 13134 webserver.cc:460] Webserver started at http://127.24.60.194:42175/ using document root <none> and password file <none>
I20260810 07:28:32.494186 13134 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:32.494262 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:32.494414 13134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:32.495534 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "cfb57763db5045b5bc50283e759bc981"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "eec71f311e537d2f18d8496f52565778"
server_key_iv: "8d09cfb7e759bfe3b8df9dbd7473948e"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.495904 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "cfb57763db5045b5bc50283e759bc981"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "eec71f311e537d2f18d8496f52565778"
server_key_iv: "8d09cfb7e759bfe3b8df9dbd7473948e"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.497298 13134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:32.498186 13150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.498384 13134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:32.498495 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "cfb57763db5045b5bc50283e759bc981"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "eec71f311e537d2f18d8496f52565778"
server_key_iv: "8d09cfb7e759bfe3b8df9dbd7473948e"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.498584 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:32.512890 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:32.513337 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:32.513516 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:32.513782 13134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:32.514143 13134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:32.514200 13134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.514243 13134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:32.514271 13134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.520393 13134 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:35755
I20260810 07:28:32.520438 13280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:35755 every 8 connection(s)
I20260810 07:28:32.520799 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:32.526490 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.521623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:37733 (local address 127.24.60.254:43099)
0810 07:28:32.521694 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.521696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.521951 (+ 255us) server_negotiation.cc:407] Connection header received
0810 07:28:32.522432 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.522434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.522446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.522511 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:32.523189 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.523688 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.524903 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.525104 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.525218 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.526212 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:32.526213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:32.526214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:32.526218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:32.526230 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:32.526233 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:32.526275 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.526346 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.526367 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:32.526681 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.521604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:43099 (local address 127.24.60.194:37733)
0810 07:28:32.521792 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.521834 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:32.522319 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:32.522571 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:32.522575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:32.522584 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:32.523105 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.523110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.523780 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.523781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.524804 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.524807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.525293 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.525294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.525418 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.526041 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:32.526041 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:32.526071 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:32.526114 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:32.526277 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:32.526281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:32.526284 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:32.526337 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 07:28:32.526455 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260810 07:28:32.527194 13281 heartbeater.cc:347] Connected to a master server at 127.24.60.254:43099
I20260810 07:28:32.527314 13281 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:32.527542 13281 heartbeater.cc:511] Master 127.24.60.254:43099 requested a full tablet report, sending...
I20260810 07:28:32.528055 12915 ts_manager.cc:194] Registered new tserver with Master: cfb57763db5045b5bc50283e759bc981 (127.24.60.194:35755)
I20260810 07:28:32.528398 12915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.194:37733
I20260810 07:28:32.529008 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 13134
I20260810 07:28:32.529120 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:32.529351 24819 external_mini_cluster.cc:1428] Setting key c4ed351b3479570532f26345787c7d52
I20260810 07:28:32.530393 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43099
--builtin_ntp_servers=127.24.60.212:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:32.615341 13284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:32.615610 13284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:32.615680 13284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:32.616922 13284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:32.616969 13284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:32.616995 13284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:32.618474 13284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:43099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:32.618763 13284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:32.619099 13284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:32.624447 13290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:32.624483 13294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:32.624688 13284 server_base.cc:1034] running on GCE node
W20260810 07:28:32.624470 13291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:32.625005 13284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:32.625310 13284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:32.626472 13284 hybrid_clock.cc:648] HybridClock initialized: now 1786346912626449 us; error 42 us; skew 500 ppm
I20260810 07:28:32.627838 13284 webserver.cc:460] Webserver started at http://127.24.60.195:43903/ using document root <none> and password file <none>
I20260810 07:28:32.628135 13284 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:32.628182 13284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:32.628350 13284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:32.629591 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "04452ca49c7d60082267619073f3cc18"
server_key_iv: "276737033eb961311963764588b3364c"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.629963 13284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "04452ca49c7d60082267619073f3cc18"
server_key_iv: "276737033eb961311963764588b3364c"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.631325 13284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:32.632094 13302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.632368 13284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:32.632472 13284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5"
format_stamp: "Formatted at 2026-08-10 07:28:32 on dist-test-slave-vxj2"
server_key: "04452ca49c7d60082267619073f3cc18"
server_key_iv: "276737033eb961311963764588b3364c"
server_key_version: "encryptionkey@0"
I20260810 07:28:32.632530 13284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:32.649022 13284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:32.649466 13284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:32.649587 13284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:32.649786 13284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:32.650162 13284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:32.650198 13284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.650220 13284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:32.650271 13284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:32.656323 13284 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:36281
I20260810 07:28:32.656425 13444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:36281 every 8 connection(s)
I20260810 07:28:32.656770 13284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:32.662611 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.657680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:36159 (local address 127.24.60.254:43099)
0810 07:28:32.657808 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.657810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.658028 (+ 218us) server_negotiation.cc:407] Connection header received
0810 07:28:32.658597 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.658599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.658612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.658670 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:32.659460 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.659982 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.661026 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.661305 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.661379 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.662297 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:32.662298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:32.662298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:32.662303 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:32.662319 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:32.662323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:32.662392 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.662495 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.662526 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:28:32.662931 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.657630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:43099 (local address 127.24.60.195:36159)
0810 07:28:32.657895 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.657930 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:32.658484 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:32.658760 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:32.658764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:32.658772 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:32.659353 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.659359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.660087 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.660088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.660916 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.660919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.661410 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.661411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.661544 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.662132 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:32.662132 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:32.662162 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:32.662187 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:32.662393 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:32.662401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:32.662404 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:32.662498 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 07:28:32.662604 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260810 07:28:32.663559 13445 heartbeater.cc:347] Connected to a master server at 127.24.60.254:43099
I20260810 07:28:32.663689 13445 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:32.663894 13445 heartbeater.cc:511] Master 127.24.60.254:43099 requested a full tablet report, sending...
I20260810 07:28:32.664348 12915 ts_manager.cc:194] Registered new tserver with Master: 2bb98b9cbce846d3aaf19cda5d6f24c5 (127.24.60.195:36281)
I20260810 07:28:32.664716 12915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.195:36159
I20260810 07:28:32.665999 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 13284
I20260810 07:28:32.666122 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:32.666343 24819 external_mini_cluster.cc:1428] Setting key 2e6f068eb6574a22084d4bba59d9e632
I20260810 07:28:32.667325 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:32.671787 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.668876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.24.60.254:43099)
0810 07:28:32.668938 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.668942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.669068 (+ 126us) server_negotiation.cc:407] Connection header received
0810 07:28:32.669108 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.669109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.669117 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.669225 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:32.669378 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.669850 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.670670 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.670851 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.670905 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.671563 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:32.671564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:32.671564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:32.671569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:32.671591 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:32.671596 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:32.671634 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.671703 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.671726 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:28:32.673391 12915 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:32.674906 12915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:32.681617 13453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.677232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36281 (local address 127.0.0.1:57170)
0810 07:28:32.677380 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.677430 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:32.677521 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:32.678147 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:32.678150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:32.678153 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:32.678226 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.678228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.679179 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.679180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.680344 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.680347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.681295 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.681296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.681384 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.681402 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 07:28:32.681469 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:28:32.681525 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260810 07:28:32.681717 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.677324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.24.60.193:46843)
0810 07:28:32.677891 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.677898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.677910 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:28:32.677976 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.677979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.678004 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.678321 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:32.678394 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.679505 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.680713 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.681324 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.681422 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.681525 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.681588 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.681626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":137,"threads_started":1}
I20260810 07:28:32.681748 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.677232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46843 (local address 127.0.0.1:56144)
0810 07:28:32.677569 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.677579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:32.677634 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:32.678152 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:32.678153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:32.678155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:32.678237 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.678239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.679603 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.679604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.680631 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.680633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.681446 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.681447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.681518 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.681522 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:32.681578 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 07:28:32.681623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":24,"spinlock_wait_cycles":384,"thread_start_us":90,"threads_started":1}
I20260810 07:28:32.681988 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.677659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.24.60.195:36281)
0810 07:28:32.677872 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.677877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.677887 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:32.677931 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.677933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.677949 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.678034 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:32.678367 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.679089 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.680431 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.681195 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.681265 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.681385 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.681884 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.681921 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260810 07:28:32.682367 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.677490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:35755 (local address 127.0.0.1:47734)
0810 07:28:32.677870 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.677881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:32.678021 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:32.678223 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:32.678224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:32.678225 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:32.678260 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.678261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.679624 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.679625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.680625 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:32.680627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.681638 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.681639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:32.682206 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.682211 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:28:32.682266 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 07:28:32.682306 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":88,"threads_started":1}
I20260810 07:28:32.682610 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.677629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.24.60.194:35755)
0810 07:28:32.677872 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:32.677877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:32.677967 (+ 90us) server_negotiation.cc:407] Connection header received
0810 07:28:32.678132 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:32.678135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:32.678150 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:32.678230 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:32.678316 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.679533 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.680723 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:32.681462 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:32.681525 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:32.681605 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:32.682477 (+ 872us) server_negotiation.cc:299] Negotiation successful
0810 07:28:32.682521 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
I20260810 07:28:32.682565 13052 tablet_service.cc:1467] Processing CreateTablet for tablet 931081e0a4984df6a759a29a52bbb11b (DEFAULT_TABLE table=test-table [id=678466e194734f828633789a9fc796e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:32.682682 13361 tablet_service.cc:1467] Processing CreateTablet for tablet 931081e0a4984df6a759a29a52bbb11b (DEFAULT_TABLE table=test-table [id=678466e194734f828633789a9fc796e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:32.682981 13052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931081e0a4984df6a759a29a52bbb11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:32.683030 13361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931081e0a4984df6a759a29a52bbb11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:32.683270 13202 tablet_service.cc:1467] Processing CreateTablet for tablet 931081e0a4984df6a759a29a52bbb11b (DEFAULT_TABLE table=test-table [id=678466e194734f828633789a9fc796e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:32.683586 13202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931081e0a4984df6a759a29a52bbb11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:32.685911 13462 tablet_bootstrap.cc:492] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Bootstrap starting.
I20260810 07:28:32.686551 13463 tablet_bootstrap.cc:492] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: Bootstrap starting.
I20260810 07:28:32.687075 13462 tablet_bootstrap.cc:654] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:32.687103 13464 tablet_bootstrap.cc:492] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Bootstrap starting.
I20260810 07:28:32.687218 13463 tablet_bootstrap.cc:654] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:32.687443 13462 log.cc:824] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:32.687615 13463 log.cc:824] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:32.687819 13464 tablet_bootstrap.cc:654] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:32.688184 13464 log.cc:824] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:32.688295 13462 tablet_bootstrap.cc:492] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: No bootstrap required, opened a new log
I20260810 07:28:32.688369 13462 ts_tablet_manager.cc:1397] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:28:32.688424 13463 tablet_bootstrap.cc:492] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: No bootstrap required, opened a new log
I20260810 07:28:32.688495 13463 ts_tablet_manager.cc:1397] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:28:32.689075 13464 tablet_bootstrap.cc:492] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: No bootstrap required, opened a new log
I20260810 07:28:32.689182 13464 ts_tablet_manager.cc:1397] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:32.690596 13462 raft_consensus.cc:348] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:32.690757 13462 raft_consensus.cc:374] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:32.690802 13462 raft_consensus.cc:729] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb98b9cbce846d3aaf19cda5d6f24c5, State: Initialized, Role: FOLLOWER
I20260810 07:28:32.690913 13462 consensus_queue.cc:260] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:32.691215 13462 ts_tablet_manager.cc:1428] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:32.691231 13445 heartbeater.cc:503] Master 127.24.60.254:43099 was elected leader, sending a full tablet report...
I20260810 07:28:32.691236 13463 raft_consensus.cc:348] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:32.691383 13463 raft_consensus.cc:374] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:32.691421 13463 raft_consensus.cc:729] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c529d166594994bbaff974fb49b7e3, State: Initialized, Role: FOLLOWER
I20260810 07:28:32.691561 13463 consensus_queue.cc:260] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:32.691861 13130 heartbeater.cc:503] Master 127.24.60.254:43099 was elected leader, sending a full tablet report...
I20260810 07:28:32.691874 13464 raft_consensus.cc:348] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:32.692016 13464 raft_consensus.cc:374] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:32.692049 13464 raft_consensus.cc:729] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb57763db5045b5bc50283e759bc981, State: Initialized, Role: FOLLOWER
I20260810 07:28:32.692219 13463 ts_tablet_manager.cc:1428] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 07:28:32.692183 13464 consensus_queue.cc:260] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:32.692499 13464 ts_tablet_manager.cc:1428] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:32.692622 13281 heartbeater.cc:503] Master 127.24.60.254:43099 was elected leader, sending a full tablet report...
W20260810 07:28:32.771359 13282 tablet.cc:2369] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:32.891312 13131 tablet.cc:2369] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:32.907426 13446 tablet.cc:2369] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:33.011438 13472 raft_consensus.cc:482] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:33.011590 13472 raft_consensus.cc:504] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:33.011961 13472 leader_election.cc:288] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb98b9cbce846d3aaf19cda5d6f24c5 (127.24.60.195:36281), 15c529d166594994bbaff974fb49b7e3 (127.24.60.193:46843)
I20260810 07:28:33.016417 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:33.012308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:59019 (local address 127.24.60.195:36281)
0810 07:28:33.012359 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:33.012361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:33.012376 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:28:33.012455 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:33.012456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:33.012464 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:33.012565 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:33.012813 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.013616 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.015409 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.015981 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.016059 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:33.016068 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:33.016289 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:28:33.016335 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:28:33.016417 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:33.012322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:49719 (local address 127.24.60.193:46843)
0810 07:28:33.012387 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:33.012389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:33.012521 (+ 132us) server_negotiation.cc:407] Connection header received
0810 07:28:33.012554 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:33.012555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:33.012568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:33.012645 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:33.012809 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.013640 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.015408 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.015976 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.016059 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:33.016068 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:33.016288 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 07:28:33.016334 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:33.016451 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:33.012219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:36281 (local address 127.24.60.194:59019)
0810 07:28:33.012307 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:33.012322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:33.012376 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:33.012595 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:33.012596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:33.012597 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:33.012717 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:33.012720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.013751 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.013753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:33.015280 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:33.015285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.016098 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.016100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:33.016190 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:33.016198 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 07:28:33.016271 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:28:33.016331 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":44}
I20260810 07:28:33.016458 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:33.012219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46843 (local address 127.24.60.194:49719)
0810 07:28:33.012434 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:33.012444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:33.012507 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:33.012657 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:33.012658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:33.012659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:33.012727 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:33.012729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.013752 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.013753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:33.015280 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:33.015285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.016113 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.016114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:33.016190 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:33.016198 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 07:28:33.016271 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:28:33.016331 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":111,"thread_start_us":102,"threads_started":1}
I20260810 07:28:33.016968 13388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931081e0a4984df6a759a29a52bbb11b" candidate_uuid: "cfb57763db5045b5bc50283e759bc981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" is_pre_election: true
I20260810 07:28:33.016965 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931081e0a4984df6a759a29a52bbb11b" candidate_uuid: "cfb57763db5045b5bc50283e759bc981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c529d166594994bbaff974fb49b7e3" is_pre_election: true
I20260810 07:28:33.017182 13388 raft_consensus.cc:2454] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfb57763db5045b5bc50283e759bc981 in term 0.
I20260810 07:28:33.017182 13076 raft_consensus.cc:2454] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfb57763db5045b5bc50283e759bc981 in term 0.
I20260810 07:28:33.017496 13151 leader_election.cc:302] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c529d166594994bbaff974fb49b7e3, cfb57763db5045b5bc50283e759bc981; no voters:
I20260810 07:28:33.017727 13472 raft_consensus.cc:2781] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:33.017817 13472 raft_consensus.cc:482] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:33.017850 13472 raft_consensus.cc:3037] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:33.018630 13472 raft_consensus.cc:504] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:33.018797 13472 leader_election.cc:288] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb98b9cbce846d3aaf19cda5d6f24c5 (127.24.60.195:36281), 15c529d166594994bbaff974fb49b7e3 (127.24.60.193:46843)
I20260810 07:28:33.018961 13388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931081e0a4984df6a759a29a52bbb11b" candidate_uuid: "cfb57763db5045b5bc50283e759bc981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5"
I20260810 07:28:33.019074 13076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931081e0a4984df6a759a29a52bbb11b" candidate_uuid: "cfb57763db5045b5bc50283e759bc981" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c529d166594994bbaff974fb49b7e3"
I20260810 07:28:33.019078 13388 raft_consensus.cc:3037] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:33.019163 13076 raft_consensus.cc:3037] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:33.020125 13388 raft_consensus.cc:2454] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb57763db5045b5bc50283e759bc981 in term 1.
I20260810 07:28:33.020125 13076 raft_consensus.cc:2454] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb57763db5045b5bc50283e759bc981 in term 1.
I20260810 07:28:33.020341 13151 leader_election.cc:302] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c529d166594994bbaff974fb49b7e3, cfb57763db5045b5bc50283e759bc981; no voters:
I20260810 07:28:33.020473 13472 raft_consensus.cc:2781] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:33.020648 13472 raft_consensus.cc:686] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 1 LEADER]: Becoming Leader. State: Replica: cfb57763db5045b5bc50283e759bc981, State: Running, Role: LEADER
I20260810 07:28:33.020787 13472 consensus_queue.cc:237] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } }
I20260810 07:28:33.021802 12914 catalog_manager.cc:5495] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfb57763db5045b5bc50283e759bc981 (127.24.60.194). New cstate: current_term: 1 leader_uuid: "cfb57763db5045b5bc50283e759bc981" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfb57763db5045b5bc50283e759bc981" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 35755 } health_report { overall_health: HEALTHY } } }
I20260810 07:28:33.039645 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:33.037068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.24.60.194:35755)
0810 07:28:33.037137 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:33.037139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:33.037184 (+ 45us) server_negotiation.cc:407] Connection header received
0810 07:28:33.037235 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:33.037235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:33.037257 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:33.037345 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:33.037509 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.038000 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.038793 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:33.039007 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:33.039091 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:33.039269 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:33.039389 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:33.039494 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:33.039513 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 07:28:33.039532 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:28:33.041579 13076 raft_consensus.cc:1261] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 1 FOLLOWER]: Refusing update from remote peer cfb57763db5045b5bc50283e759bc981: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:33.041575 13388 raft_consensus.cc:1261] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 1 FOLLOWER]: Refusing update from remote peer cfb57763db5045b5bc50283e759bc981: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:33.041913 13472 consensus_queue.cc:1035] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb98b9cbce846d3aaf19cda5d6f24c5" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 36281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:33.042074 13475 consensus_queue.cc:1035] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c529d166594994bbaff974fb49b7e3" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:33.046078 13478 mvcc.cc:205] Tried to move back new op lower bound from 7316876955815489536 to 7316876955734024192. Current Snapshot: MvccSnapshot[applied={T|T < 7316876955815489536 or (T in {7316876955815489536})}]
I20260810 07:28:33.047088 12914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:33.047194 12914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:33.047993 12914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8883b92aaeab42929c9b78736ab04b53: Sending DeleteTablet for 3 replicas of tablet 931081e0a4984df6a759a29a52bbb11b
I20260810 07:28:33.048331 13361 tablet_service.cc:1514] Processing DeleteTablet for tablet 931081e0a4984df6a759a29a52bbb11b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:33 UTC) from {username='slave'} at 127.0.0.1:57170
I20260810 07:28:33.048430 13052 tablet_service.cc:1514] Processing DeleteTablet for tablet 931081e0a4984df6a759a29a52bbb11b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:33 UTC) from {username='slave'} at 127.0.0.1:56144
I20260810 07:28:33.048560 13202 tablet_service.cc:1514] Processing DeleteTablet for tablet 931081e0a4984df6a759a29a52bbb11b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:33 UTC) from {username='slave'} at 127.0.0.1:47734
I20260810 07:28:33.048655 13489 tablet_replica.cc:330] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3: stopping tablet replica
I20260810 07:28:33.048719 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12985
I20260810 07:28:33.048758 13489 raft_consensus.cc:2229] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:33.048870 13489 raft_consensus.cc:2258] T 931081e0a4984df6a759a29a52bbb11b P 15c529d166594994bbaff974fb49b7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:33.052455 13488 tablet_replica.cc:330] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: stopping tablet replica
I20260810 07:28:33.052591 13488 raft_consensus.cc:2229] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:33.052716 13488 raft_consensus.cc:2258] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:33.053357 13488 ts_tablet_manager.cc:1905] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:33.054603 13490 tablet_replica.cc:330] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: stopping tablet replica
I20260810 07:28:33.054746 13490 raft_consensus.cc:2229] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:28:33.054878 13490 raft_consensus.cc:2258] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:33.055791 13490 ts_tablet_manager.cc:1905] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:33.057897 13490 ts_tablet_manager.cc:1918] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:33.057974 13490 log.cc:1196] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/931081e0a4984df6a759a29a52bbb11b
I20260810 07:28:33.058233 13490 ts_tablet_manager.cc:1939] T 931081e0a4984df6a759a29a52bbb11b P cfb57763db5045b5bc50283e759bc981: Deleting consensus metadata
I20260810 07:28:33.058760 12899 catalog_manager.cc:4841] TS cfb57763db5045b5bc50283e759bc981 (127.24.60.194:35755): tablet 931081e0a4984df6a759a29a52bbb11b (table test-table [id=678466e194734f828633789a9fc796e0]) successfully deleted
I20260810 07:28:33.059017 13488 ts_tablet_manager.cc:1918] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:33.059098 13488 log.cc:1196] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/931081e0a4984df6a759a29a52bbb11b
I20260810 07:28:33.059365 13488 ts_tablet_manager.cc:1939] T 931081e0a4984df6a759a29a52bbb11b P 2bb98b9cbce846d3aaf19cda5d6f24c5: Deleting consensus metadata
I20260810 07:28:33.059983 12899 catalog_manager.cc:4841] TS 2bb98b9cbce846d3aaf19cda5d6f24c5 (127.24.60.195:36281): tablet 931081e0a4984df6a759a29a52bbb11b (table test-table [id=678466e194734f828633789a9fc796e0]) successfully deleted
I20260810 07:28:33.060322 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 13134
W20260810 07:28:33.059978 12897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:46843 (error 108)
I20260810 07:28:33.062124 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:33.061975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:46843 (local address 127.0.0.1:56154)
0810 07:28:33.062047 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:33.062095 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 07:28:33.062268 12897 catalog_manager.cc:4584] TS 15c529d166594994bbaff974fb49b7e3 (127.24.60.193:46843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 931081e0a4984df6a759a29a52bbb11b: Network error: Client connection negotiation failed: client connection to 127.24.60.193:46843: connect: Connection refused (error 111)
I20260810 07:28:33.069061 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 13284
I20260810 07:28:33.077101 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 12879
2026-08-10T07:28:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1013 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:33 dist-test-slave-vxj2 krb5kdc[13495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:33 dist-test-slave-vxj2 krb5kdc[13495](info): set up 2 sockets
Aug 10 07:28:33 dist-test-slave-vxj2 krb5kdc[13495](info): commencing operation
krb5kdc: starting...
W20260810 07:28:34.222440 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:28:34.355670 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34273
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:34273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:34.441015 13508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:34.441303 13508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:34.441331 13508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:34.442476 13508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:34.442514 13508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:34.442528 13508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:34.442541 13508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:34.442553 13508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:34.444007 13508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:34273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:34.444222 13508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:34.444402 13508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:34.449580 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:34.449601 13514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:34.449648 13508 server_base.cc:1034] running on GCE node
W20260810 07:28:34.449589 13513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:34.450017 13508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:34.450402 13508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:34.451618 13508 hybrid_clock.cc:648] HybridClock initialized: now 1786346914451604 us; error 47 us; skew 500 ppm
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:34.454528 13508 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:28:34.455018 13508 webserver.cc:460] Webserver started at http://127.24.60.254:34969/ using document root <none> and password file <none>
I20260810 07:28:34.455257 13508 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:34.455328 13508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:34.455494 13508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:34.456676 13508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "fc556925dcca4147807b2addedf3a0e2"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "eb7d6ac06e2c3bbee1504eb641994047"
server_key_iv: "14c45ab289b17271a58d7e511f712108"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.457063 13508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "fc556925dcca4147807b2addedf3a0e2"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "eb7d6ac06e2c3bbee1504eb641994047"
server_key_iv: "14c45ab289b17271a58d7e511f712108"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.458510 13508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:34.459267 13526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.459470 13508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:28:34.459580 13508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "fc556925dcca4147807b2addedf3a0e2"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "eb7d6ac06e2c3bbee1504eb641994047"
server_key_iv: "14c45ab289b17271a58d7e511f712108"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.459666 13508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:34.474905 13508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:34.477519 13508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:34.477698 13508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:34.481987 13508 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:34273
I20260810 07:28:34.482048 13584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:34273 every 8 connection(s)
I20260810 07:28:34.482504 13508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:34.483425 13585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:34.485867 13585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2: Bootstrap starting.
I20260810 07:28:34.486488 13585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:34.486802 13585 log.cc:824] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:34.487536 13585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2: No bootstrap required, opened a new log
I20260810 07:28:34.489198 13585 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc556925dcca4147807b2addedf3a0e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34273 } }
I20260810 07:28:34.489341 13585 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:34.489362 13585 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc556925dcca4147807b2addedf3a0e2, State: Initialized, Role: FOLLOWER
I20260810 07:28:34.489547 13585 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc556925dcca4147807b2addedf3a0e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34273 } }
I20260810 07:28:34.489614 13585 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:34.489635 13585 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:34.489706 13585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:34.490307 13585 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc556925dcca4147807b2addedf3a0e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34273 } }
I20260810 07:28:34.490423 13585 leader_election.cc:302] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc556925dcca4147807b2addedf3a0e2; no voters:
I20260810 07:28:34.490621 13585 leader_election.cc:288] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:34.490679 13588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:34.490849 13588 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [term 1 LEADER]: Becoming Leader. State: Replica: fc556925dcca4147807b2addedf3a0e2, State: Running, Role: LEADER
I20260810 07:28:34.490896 13585 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:34.490947 13588 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc556925dcca4147807b2addedf3a0e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34273 } }
I20260810 07:28:34.491430 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 13508
I20260810 07:28:34.491596 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:34.491881 24819 external_mini_cluster.cc:1428] Setting key c15740ea44061194cb7a649c6bb36a6d
I20260810 07:28:34.492014 13589 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc556925dcca4147807b2addedf3a0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc556925dcca4147807b2addedf3a0e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34273 } } }
I20260810 07:28:34.492136 13589 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:34.492027 13590 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc556925dcca4147807b2addedf3a0e2. Latest consensus state: current_term: 1 leader_uuid: "fc556925dcca4147807b2addedf3a0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc556925dcca4147807b2addedf3a0e2" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34273 } } }
I20260810 07:28:34.492233 13590 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:34.492388 13604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:34.493005 13604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:34.494613 13604 catalog_manager.cc:1347] Generated new cluster ID: 6773030e0b0b45d5a3b9a67129e687a9
I20260810 07:28:34.494685 13604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:34.503021 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:34.492614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.24.60.254:34273)
0810 07:28:34.492792 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:34.492802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:34.492838 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:28:34.492981 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:34.492989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:34.493059 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:34.493266 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:34.493913 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.495167 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.495975 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.496407 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.496532 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:34.499106 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:34.499110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:34.499111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:34.499126 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:34.501172 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:34.501619 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:34.501622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:34.501622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:34.501753 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:34.501985 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:34.501986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:34.501986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:34.502308 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:34.502452 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:34.502594 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 07:28:34.502736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260810 07:28:34.506803 13604 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:34.507359 13604 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:34.510094 13604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2: Generated new TSK 0
I20260810 07:28:34.510330 13604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:28:34.543658 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34273
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:34.630381 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:34.630597 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:34.630623 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:34.631753 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:34.631796 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:34.631822 13615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:34.633294 13615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:34.633505 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:34.633705 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:34.638911 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:34.638911 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:34.639067 13623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:34.639135 13615 server_base.cc:1034] running on GCE node
I20260810 07:28:34.639341 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:34.639623 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:34.640779 13615 hybrid_clock.cc:648] HybridClock initialized: now 1786346914640759 us; error 40 us; skew 500 ppm
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:34.643478 13615 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:28:34.643973 13615 webserver.cc:460] Webserver started at http://127.24.60.193:42555/ using document root <none> and password file <none>
I20260810 07:28:34.644222 13615 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:34.644268 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:34.644439 13615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:34.645612 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "72ace69ad9839aeebda1578eb3a2b45b"
server_key_iv: "8c99c936875238b190e6bf3d8d3c4b00"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.646059 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "72ace69ad9839aeebda1578eb3a2b45b"
server_key_iv: "8c99c936875238b190e6bf3d8d3c4b00"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.647414 13615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:34.648138 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.648335 13615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:28:34.648432 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "72ace69ad9839aeebda1578eb3a2b45b"
server_key_iv: "8c99c936875238b190e6bf3d8d3c4b00"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.648486 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:34.662072 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:34.664577 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:34.664703 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:34.664906 13615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:34.665315 13615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:34.665356 13615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.665380 13615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:34.665417 13615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.671458 13615 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:34509
I20260810 07:28:34.671552 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:34509 every 8 connection(s)
I20260810 07:28:34.671901 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:34.679533 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 13615
I20260810 07:28:34.679678 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:34.679931 24819 external_mini_cluster.cc:1428] Setting key 5886ccb0f3a9b0c4978b7da499889e71
I20260810 07:28:34.682174 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:34.672817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:53009 (local address 127.24.60.254:34273)
0810 07:28:34.672887 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:34.672889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:34.673075 (+ 186us) server_negotiation.cc:407] Connection header received
0810 07:28:34.673599 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:34.673600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:34.673610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:34.673675 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:34.674611 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.675116 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.676094 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.676294 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.676346 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:34.678608 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:34.678609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:34.678610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:34.678617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:34.680417 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:34.681013 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:34.681014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:34.681014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:34.681084 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:34.681411 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:34.681412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:34.681413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:34.681575 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:34.681650 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:34.681915 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 07:28:34.682028 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:28:34.682199 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:34.672767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34273 (local address 127.24.60.193:53009)
0810 07:28:34.672937 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:34.672987 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:34.673507 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:34.673728 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:34.673731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:34.674319 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:34.674528 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:34.674532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.675204 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.675205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:34.676015 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:34.676017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.676370 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.676371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:34.676480 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:34.677075 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:34.677083 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:34.678512 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:34.680497 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:34.680506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:34.680508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:34.680884 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:34.681192 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:34.681193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:34.681194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:34.681313 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:34.681643 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:34.681649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:34.681772 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 07:28:34.681852 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:28:34.682021 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260810 07:28:34.682868 13770 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34273
I20260810 07:28:34.682971 13770 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:34.683187 13770 heartbeater.cc:511] Master 127.24.60.254:34273 requested a full tablet report, sending...
I20260810 07:28:34.683768 13546 ts_manager.cc:194] Registered new tserver with Master: 5d2d06bd0dc544dfadbc2b5a10e5c032 (127.24.60.193:34509)
I20260810 07:28:34.684746 13546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:53009
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:28:34.708603 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34273
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:34.794751 13777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:34.794977 13777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:34.795008 13777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:34.796177 13777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:34.796216 13777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:34.796242 13777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:34.797757 13777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:34.797972 13777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:34.798172 13777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:34.803541 13783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:34.803572 13784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:34.803541 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:34.803928 13777 server_base.cc:1034] running on GCE node
I20260810 07:28:34.804167 13777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:34.804428 13777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:34.805608 13777 hybrid_clock.cc:648] HybridClock initialized: now 1786346914805595 us; error 29 us; skew 500 ppm
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:34.813503 13777 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:28:34.814112 13777 webserver.cc:460] Webserver started at http://127.24.60.194:39651/ using document root <none> and password file <none>
I20260810 07:28:34.814360 13777 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:34.814432 13777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:34.814597 13777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:34.815806 13777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "03783289474dc064e17fbf8064bfc024"
server_key_iv: "96778fa23a3faeac15753b6df1793828"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.816190 13777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "03783289474dc064e17fbf8064bfc024"
server_key_iv: "96778fa23a3faeac15753b6df1793828"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.817709 13777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:34.818534 13793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.818718 13777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:34.818830 13777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "03783289474dc064e17fbf8064bfc024"
server_key_iv: "96778fa23a3faeac15753b6df1793828"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.818907 13777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:34.836620 13777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:34.839704 13777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:34.839829 13777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:34.840044 13777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:34.840313 13777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:34.840346 13777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.840371 13777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:34.840385 13777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.846601 13777 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:34529
I20260810 07:28:34.846637 13922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:34529 every 8 connection(s)
I20260810 07:28:34.846998 13777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:34.854722 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 13777
I20260810 07:28:34.854871 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:34.855108 24819 external_mini_cluster.cc:1428] Setting key 295218a36d67ea4ecb5595aa4e95ea0e
I20260810 07:28:34.857460 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:34.847865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:33255 (local address 127.24.60.254:34273)
0810 07:28:34.847973 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:34.847976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:34.848153 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:28:34.848736 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:34.848738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:34.848751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:34.848803 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:34.849897 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.850411 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.851464 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.851661 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.851741 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:34.853989 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:34.853991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:34.853991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:34.854005 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:34.855842 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:34.856391 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:34.856392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:34.856393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:34.856452 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:34.856753 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:34.856753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:34.856753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:34.856860 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:34.856926 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:34.857288 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 07:28:34.857390 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:28:34.857764 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:34.847826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34273 (local address 127.24.60.194:33255)
0810 07:28:34.848016 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:34.848067 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:34.848616 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:34.848878 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:34.848883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:34.849572 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:34.849779 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:34.849784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.850517 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.850519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:34.851343 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:34.851345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:34.851749 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:34.851750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:34.851880 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:34.852389 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:34.852398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:34.853884 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:34.855929 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:34.855936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:34.855938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:34.856305 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:34.856523 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:34.856524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:34.856524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:34.856688 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:34.856931 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:34.856937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:34.857120 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 07:28:34.857265 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 07:28:34.857469 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260810 07:28:34.858294 13923 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34273
I20260810 07:28:34.858384 13923 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:34.858593 13923 heartbeater.cc:511] Master 127.24.60.254:34273 requested a full tablet report, sending...
I20260810 07:28:34.859016 13546 ts_manager.cc:194] Registered new tserver with Master: 4ad9290cf4e94c0e8865f1e1dfc0417d (127.24.60.194:34529)
I20260810 07:28:34.859397 13546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:33255
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:28:34.884687 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34273
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:34.970746 13928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:34.970989 13928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:34.971016 13928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:34.972167 13928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:34.972203 13928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:34.972229 13928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:34.973803 13928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:34273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:34.974020 13928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:34.974202 13928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:34.979302 13933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:34.979373 13928 server_base.cc:1034] running on GCE node
W20260810 07:28:34.979282 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:34.979283 13939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:34.979769 13928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:34.980051 13928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:34.981241 13928 hybrid_clock.cc:648] HybridClock initialized: now 1786346914981221 us; error 27 us; skew 500 ppm
Aug 10 07:28:34 dist-test-slave-vxj2 krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:34.983983 13928 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:28:34.984458 13928 webserver.cc:460] Webserver started at http://127.24.60.195:44369/ using document root <none> and password file <none>
I20260810 07:28:34.984710 13928 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:34.984782 13928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:34.984942 13928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:34.986188 13928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "9c30395168304d638c93fd24cbe288d4"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "1005b45a16892f89b731e1bd123a8187"
server_key_iv: "4058ebb511847665a6c606aa1c4133d8"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.986594 13928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "9c30395168304d638c93fd24cbe288d4"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "1005b45a16892f89b731e1bd123a8187"
server_key_iv: "4058ebb511847665a6c606aa1c4133d8"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.987998 13928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:34.988765 13947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:34.988945 13928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:28:34.989058 13928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "9c30395168304d638c93fd24cbe288d4"
format_stamp: "Formatted at 2026-08-10 07:28:34 on dist-test-slave-vxj2"
server_key: "1005b45a16892f89b731e1bd123a8187"
server_key_iv: "4058ebb511847665a6c606aa1c4133d8"
server_key_version: "encryptionkey@0"
I20260810 07:28:34.989135 13928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:35.001892 13928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:35.004390 13928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:35.004573 13928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:35.004823 13928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:35.005223 13928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:35.005286 13928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.005332 13928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:35.005373 13928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.011603 13928 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:42099
I20260810 07:28:35.011699 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:42099 every 8 connection(s)
I20260810 07:28:35.012082 13928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:28:35 dist-test-slave-vxj2 krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346914, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:35.020658 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 13928
I20260810 07:28:35.020758 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:35.020973 24819 external_mini_cluster.cc:1428] Setting key 3a2f9e703ca305a39d1bcb973810abad
I20260810 07:28:35.022742 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.012857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:44543 (local address 127.24.60.254:34273)
0810 07:28:35.012929 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.012931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.013222 (+ 291us) server_negotiation.cc:407] Connection header received
0810 07:28:35.013761 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.013762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.013771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.013833 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.014862 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.015393 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.016441 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.016628 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.016687 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.019252 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.019254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.019254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:35.019285 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.020872 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:35.021557 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:35.021558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:35.021559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:35.021635 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:35.021993 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:35.021994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:35.021995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:35.022153 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.022233 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.022552 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.022649 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:28:35.022990 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.012835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:34273 (local address 127.24.60.195:44543)
0810 07:28:35.013040 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.013086 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.013667 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.013899 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.013903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.014532 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:35.014748 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.014754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.015504 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.015505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.016354 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.016357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.016711 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.016712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.016822 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.017420 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:35.017429 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:35.019102 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:35.020969 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:35.020978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:35.020981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:35.021415 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:35.021724 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:35.021725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:35.021725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:35.021847 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:35.022234 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:35.022243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:35.022398 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.022512 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.022691 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260810 07:28:35.023497 14076 heartbeater.cc:347] Connected to a master server at 127.24.60.254:34273
I20260810 07:28:35.023617 14076 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:35.023805 14076 heartbeater.cc:511] Master 127.24.60.254:34273 requested a full tablet report, sending...
I20260810 07:28:35.024173 13546 ts_manager.cc:194] Registered new tserver with Master: 9c30395168304d638c93fd24cbe288d4 (127.24.60.195:42099)
I20260810 07:28:35.024554 13546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:44543
I20260810 07:28:35.033103 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:35.040709 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.034601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.24.60.254:34273)
0810 07:28:35.034650 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.034652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.034765 (+ 113us) server_negotiation.cc:407] Connection header received
0810 07:28:35.034809 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.034810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.034819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.034865 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.035397 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.035877 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.036598 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.036779 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.036831 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.038198 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.038199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.038200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:35.038206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.039651 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:35.040025 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:35.040026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:35.040026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:35.040082 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:35.040264 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:35.040265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:35.040265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:35.040399 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.040482 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.040591 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.040641 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:28:35.042441 13546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:35.044026 13546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:35.051168 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.046909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.24.60.194:34529)
0810 07:28:35.047123 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.047128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.047183 (+ 55us) server_negotiation.cc:407] Connection header received
0810 07:28:35.047264 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.047266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.047288 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.047376 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.047730 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.048450 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.049910 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.050653 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.050737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.050799 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.051023 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.051066 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
I20260810 07:28:35.051160 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.046627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:34509 (local address 127.0.0.1:58258)
0810 07:28:35.046810 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.046853 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.046919 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.047403 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.047404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.047409 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.047625 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.047628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.048965 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.048966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.050012 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.050014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.050843 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.050845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.050922 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.050927 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.050972 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.051037 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":153,"thread_start_us":54,"threads_started":1}
I20260810 07:28:35.051163 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.046644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:34529 (local address 127.0.0.1:51952)
0810 07:28:35.047091 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.047102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.047190 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.047390 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.047394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.047396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.047636 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.047639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.048575 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.048576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.049820 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.049823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.050748 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.050749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.050830 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.050845 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.051003 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.051054 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":7,"thread_start_us":110,"threads_started":1}
I20260810 07:28:35.051595 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.046727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.24.60.193:34509)
0810 07:28:35.047222 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.047227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.047237 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:35.047278 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.047280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.047296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.047443 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.047719 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.048849 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.050106 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.050740 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.051371 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.051442 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.051468 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.051513 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":106,"threads_started":1}
I20260810 07:28:35.052306 13849 tablet_service.cc:1467] Processing CreateTablet for tablet 709770aa6b064e559b75676527c43b9c (DEFAULT_TABLE table=test-table [id=96253e20f11d49d2985f26dea44c1abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:35.052304 13694 tablet_service.cc:1467] Processing CreateTablet for tablet 709770aa6b064e559b75676527c43b9c (DEFAULT_TABLE table=test-table [id=96253e20f11d49d2985f26dea44c1abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:35.052668 13849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709770aa6b064e559b75676527c43b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.052680 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709770aa6b064e559b75676527c43b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.055569 14096 tablet_bootstrap.cc:492] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Bootstrap starting.
I20260810 07:28:35.055578 14098 tablet_bootstrap.cc:492] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032: Bootstrap starting.
I20260810 07:28:35.056290 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.046644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:42099 (local address 127.0.0.1:36388)
0810 07:28:35.047045 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.047056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.047098 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.047403 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.047404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.047406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.047627 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.047628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.049230 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.049231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.050336 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.050339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.055937 (+ 5598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.055939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.056027 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.056032 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.056106 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.056171 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":154,"thread_start_us":109,"threads_started":1}
I20260810 07:28:35.056295 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.046905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.24.60.195:42099)
0810 07:28:35.047222 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.047227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.047237 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:35.047278 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.047280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.047296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.047432 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.047717 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.048842 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.050433 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.051232 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.055931 (+ 4699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.056040 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.056133 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.056191 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260810 07:28:35.056332 14098 tablet_bootstrap.cc:654] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.056680 14096 tablet_bootstrap.cc:654] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.056998 14098 log.cc:824] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.057314 14096 log.cc:824] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.057235 14001 tablet_service.cc:1467] Processing CreateTablet for tablet 709770aa6b064e559b75676527c43b9c (DEFAULT_TABLE table=test-table [id=96253e20f11d49d2985f26dea44c1abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:35.057632 14001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709770aa6b064e559b75676527c43b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.057900 14098 tablet_bootstrap.cc:492] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032: No bootstrap required, opened a new log
I20260810 07:28:35.057981 14098 ts_tablet_manager.cc:1397] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:35.058175 14096 tablet_bootstrap.cc:492] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: No bootstrap required, opened a new log
I20260810 07:28:35.058243 14096 ts_tablet_manager.cc:1397] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:28:35.060091 14098 raft_consensus.cc:348] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.060284 14098 raft_consensus.cc:374] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.060324 14098 raft_consensus.cc:729] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2d06bd0dc544dfadbc2b5a10e5c032, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.060372 14096 raft_consensus.cc:348] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.060542 14096 raft_consensus.cc:374] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.060591 14096 raft_consensus.cc:729] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad9290cf4e94c0e8865f1e1dfc0417d, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.060490 14098 consensus_queue.cc:260] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.060710 14096 consensus_queue.cc:260] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.060952 14098 ts_tablet_manager.cc:1428] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:35.060984 14096 ts_tablet_manager.cc:1428] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:35.061002 13770 heartbeater.cc:503] Master 127.24.60.254:34273 was elected leader, sending a full tablet report...
I20260810 07:28:35.061188 14101 tablet_bootstrap.cc:492] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Bootstrap starting.
I20260810 07:28:35.061340 13923 heartbeater.cc:503] Master 127.24.60.254:34273 was elected leader, sending a full tablet report...
I20260810 07:28:35.062094 14101 tablet_bootstrap.cc:654] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.062747 14101 log.cc:824] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.063901 14101 tablet_bootstrap.cc:492] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: No bootstrap required, opened a new log
I20260810 07:28:35.064024 14101 ts_tablet_manager.cc:1397] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:28:35.065920 14101 raft_consensus.cc:348] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.066072 14101 raft_consensus.cc:374] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.066097 14101 raft_consensus.cc:729] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c30395168304d638c93fd24cbe288d4, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.066236 14101 consensus_queue.cc:260] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.066557 14101 ts_tablet_manager.cc:1428] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 07:28:35.066561 14076 heartbeater.cc:503] Master 127.24.60.254:34273 was elected leader, sending a full tablet report...
I20260810 07:28:35.094789 14102 raft_consensus.cc:482] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:35.094962 14102 raft_consensus.cc:504] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.095386 14102 leader_election.cc:288] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c30395168304d638c93fd24cbe288d4 (127.24.60.195:42099), 4ad9290cf4e94c0e8865f1e1dfc0417d (127.24.60.194:34529)
W20260810 07:28:35.097589 13924 tablet.cc:2369] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:35.098666 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.095862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:38459 (local address 127.24.60.195:42099)
0810 07:28:35.095935 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.095937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.095947 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:35.095992 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.095993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.096002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.096077 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.096251 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.096724 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.097914 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.098281 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.098377 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.098387 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.098549 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.098588 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:28:35.098666 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.095851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:58009 (local address 127.24.60.194:34529)
0810 07:28:35.095932 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.095934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.095944 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:35.095980 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.095981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.095988 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.096059 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.096238 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.096730 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.097873 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.098278 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.098378 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.098387 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.098547 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.098592 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:28:35.098713 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.095590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:34529 (local address 127.24.60.193:58009)
0810 07:28:35.095682 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.095695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.095754 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.096069 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.096070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.096072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.096155 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.096157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.096821 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.096822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.097775 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.097778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.098377 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.098378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.098467 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.098470 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.098537 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.098582 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":33}
I20260810 07:28:35.099007 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.095589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:42099 (local address 127.24.60.193:38459)
0810 07:28:35.095815 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.095829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.095885 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.096090 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.096091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.096093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.096174 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.096175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.096811 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.096812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.097738 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.097741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.098376 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.098376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.098463 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.098467 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.098539 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.098608 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":47,"thread_start_us":156,"threads_started":1}
I20260810 07:28:35.099198 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709770aa6b064e559b75676527c43b9c" candidate_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" is_pre_election: true
I20260810 07:28:35.099359 13871 raft_consensus.cc:2454] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2d06bd0dc544dfadbc2b5a10e5c032 in term 0.
I20260810 07:28:35.099447 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709770aa6b064e559b75676527c43b9c" candidate_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c30395168304d638c93fd24cbe288d4" is_pre_election: true
I20260810 07:28:35.099623 14026 raft_consensus.cc:2454] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2d06bd0dc544dfadbc2b5a10e5c032 in term 0.
I20260810 07:28:35.099624 13637 leader_election.cc:302] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad9290cf4e94c0e8865f1e1dfc0417d, 5d2d06bd0dc544dfadbc2b5a10e5c032; no voters:
I20260810 07:28:35.099823 14102 raft_consensus.cc:2781] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:35.099890 14102 raft_consensus.cc:482] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:35.099915 14102 raft_consensus.cc:3037] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.100670 14102 raft_consensus.cc:504] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.100819 14102 leader_election.cc:288] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [CANDIDATE]: Term 1 election: Requested vote from peers 9c30395168304d638c93fd24cbe288d4 (127.24.60.195:42099), 4ad9290cf4e94c0e8865f1e1dfc0417d (127.24.60.194:34529)
I20260810 07:28:35.100958 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709770aa6b064e559b75676527c43b9c" candidate_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c30395168304d638c93fd24cbe288d4"
I20260810 07:28:35.100988 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709770aa6b064e559b75676527c43b9c" candidate_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d"
I20260810 07:28:35.101035 14026 raft_consensus.cc:3037] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.101104 13871 raft_consensus.cc:3037] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.101807 14026 raft_consensus.cc:2454] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2d06bd0dc544dfadbc2b5a10e5c032 in term 1.
I20260810 07:28:35.101938 13871 raft_consensus.cc:2454] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2d06bd0dc544dfadbc2b5a10e5c032 in term 1.
I20260810 07:28:35.102062 13639 leader_election.cc:302] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2d06bd0dc544dfadbc2b5a10e5c032, 9c30395168304d638c93fd24cbe288d4; no voters:
I20260810 07:28:35.102165 14102 raft_consensus.cc:2781] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:35.102312 14102 raft_consensus.cc:686] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2d06bd0dc544dfadbc2b5a10e5c032, State: Running, Role: LEADER
I20260810 07:28:35.102409 14102 consensus_queue.cc:237] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } }
I20260810 07:28:35.103250 13544 catalog_manager.cc:5495] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d2d06bd0dc544dfadbc2b5a10e5c032 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d2d06bd0dc544dfadbc2b5a10e5c032" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 34509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 } health_report { overall_health: UNKNOWN } } }
I20260810 07:28:35.109403 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.107144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.24.60.193:34509)
0810 07:28:35.107204 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.107206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.107214 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:28:35.107267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.107268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.107284 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.107344 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:35.107469 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.107951 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.108645 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.108826 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.108874 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.109019 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:35.109106 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:35.109244 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.109268 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.109279 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:28:35.111249 14026 raft_consensus.cc:1261] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 1 FOLLOWER]: Refusing update from remote peer 5d2d06bd0dc544dfadbc2b5a10e5c032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:35.111284 13871 raft_consensus.cc:1261] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 1 FOLLOWER]: Refusing update from remote peer 5d2d06bd0dc544dfadbc2b5a10e5c032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:35.111716 14102 consensus_queue.cc:1035] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad9290cf4e94c0e8865f1e1dfc0417d" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:35.111899 14107 consensus_queue.cc:1035] T 709770aa6b064e559b75676527c43b9c P 5d2d06bd0dc544dfadbc2b5a10e5c032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c30395168304d638c93fd24cbe288d4" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:35.114007 14112 mvcc.cc:205] Tried to move back new op lower bound from 7316876964292857856 to 7316876964259983360. Current Snapshot: MvccSnapshot[applied={T|T < 7316876964292857856}]
I20260810 07:28:35.114439 14114 mvcc.cc:205] Tried to move back new op lower bound from 7316876964292857856 to 7316876964259983360. Current Snapshot: MvccSnapshot[applied={T|T < 7316876964292857856}]
I20260810 07:28:35.114634 14113 mvcc.cc:205] Tried to move back new op lower bound from 7316876964292857856 to 7316876964259983360. Current Snapshot: MvccSnapshot[applied={T|T < 7316876964292857856}]
I20260810 07:28:35.116768 13544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:35.116875 13544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:35.117691 13544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc556925dcca4147807b2addedf3a0e2: Sending DeleteTablet for 3 replicas of tablet 709770aa6b064e559b75676527c43b9c
I20260810 07:28:35.118057 14001 tablet_service.cc:1514] Processing DeleteTablet for tablet 709770aa6b064e559b75676527c43b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:35 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:36388
I20260810 07:28:35.118088 13849 tablet_service.cc:1514] Processing DeleteTablet for tablet 709770aa6b064e559b75676527c43b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:35 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:51952
I20260810 07:28:35.118287 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 709770aa6b064e559b75676527c43b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:35 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:58258
I20260810 07:28:35.118315 14121 tablet_replica.cc:330] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: stopping tablet replica
I20260810 07:28:35.118417 14121 raft_consensus.cc:2229] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:35.118527 14121 raft_consensus.cc:2258] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:35.118571 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 13615
I20260810 07:28:35.118693 14122 tablet_replica.cc:330] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: stopping tablet replica
I20260810 07:28:35.118794 14122 raft_consensus.cc:2229] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:35.118891 14122 raft_consensus.cc:2258] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:35.119453 14121 ts_tablet_manager.cc:1905] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:35.121243 14121 ts_tablet_manager.cc:1918] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:35.121322 14121 log.cc:1196] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/709770aa6b064e559b75676527c43b9c
I20260810 07:28:35.121639 14121 ts_tablet_manager.cc:1939] T 709770aa6b064e559b75676527c43b9c P 9c30395168304d638c93fd24cbe288d4: Deleting consensus metadata
I20260810 07:28:35.121731 14122 ts_tablet_manager.cc:1905] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:35.122048 13529 catalog_manager.cc:4841] TS 9c30395168304d638c93fd24cbe288d4 (127.24.60.195:42099): tablet 709770aa6b064e559b75676527c43b9c (table test-table [id=96253e20f11d49d2985f26dea44c1abf]) successfully deleted
I20260810 07:28:35.123903 14122 ts_tablet_manager.cc:1918] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:35.123971 14122 log.cc:1196] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/709770aa6b064e559b75676527c43b9c
I20260810 07:28:35.124251 14122 ts_tablet_manager.cc:1939] T 709770aa6b064e559b75676527c43b9c P 4ad9290cf4e94c0e8865f1e1dfc0417d: Deleting consensus metadata
I20260810 07:28:35.124683 13527 catalog_manager.cc:4841] TS 4ad9290cf4e94c0e8865f1e1dfc0417d (127.24.60.194:34529): tablet 709770aa6b064e559b75676527c43b9c (table test-table [id=96253e20f11d49d2985f26dea44c1abf]) successfully deleted
W20260810 07:28:35.127282 13530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:34509 (error 108)
I20260810 07:28:35.127465 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 13777
I20260810 07:28:35.130910 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.130706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:34509 (local address 127.0.0.1:58278)
0810 07:28:35.130792 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.130854 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:34509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 07:28:35.131124 13530 catalog_manager.cc:4584] TS 5d2d06bd0dc544dfadbc2b5a10e5c032 (127.24.60.193:34509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 709770aa6b064e559b75676527c43b9c: Network error: Client connection negotiation failed: client connection to 127.24.60.193:34509: connect: Connection refused (error 111)
I20260810 07:28:35.135423 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 13928
I20260810 07:28:35.143649 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 13508
2026-08-10T07:28:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T07:28:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:35Z Disabled control of system clock
I20260810 07:28:35.175923 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:40355
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:40355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:35.260609 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:35.260852 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:35.260875 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:35.262071 14128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:35.262115 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:35.262127 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:35.262140 14128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:35.262152 14128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:35.263491 14128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:40355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:40355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:35.263706 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:35.263975 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:35.268908 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.268913 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.268994 14141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:35.269202 14128 server_base.cc:1034] running on GCE node
I20260810 07:28:35.269496 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:35.269788 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:35.270959 14128 hybrid_clock.cc:648] HybridClock initialized: now 1786346915270942 us; error 49 us; skew 500 ppm
I20260810 07:28:35.272203 14128 webserver.cc:460] Webserver started at http://127.24.60.254:35107/ using document root <none> and password file <none>
I20260810 07:28:35.272508 14128 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:35.272583 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:35.272742 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:35.273967 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "cc89b5beac5c4e729e14089414cc44f0"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "f12f9ba9c7f631d09be8c2c3e2e06d6e"
server_key_iv: "2099c254eb0d16d8033b0f7719087f7e"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.274338 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "cc89b5beac5c4e729e14089414cc44f0"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "f12f9ba9c7f631d09be8c2c3e2e06d6e"
server_key_iv: "2099c254eb0d16d8033b0f7719087f7e"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.275719 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:35.276528 14147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.276717 14128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:35.276830 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "cc89b5beac5c4e729e14089414cc44f0"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "f12f9ba9c7f631d09be8c2c3e2e06d6e"
server_key_iv: "2099c254eb0d16d8033b0f7719087f7e"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.276911 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:35.287585 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:35.287985 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:35.288146 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:35.292296 14128 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:40355
I20260810 07:28:35.292335 14206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:40355 every 8 connection(s)
I20260810 07:28:35.292790 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:35.293865 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.296224 14207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0: Bootstrap starting.
I20260810 07:28:35.296830 14207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.297192 14207 log.cc:824] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.297984 14207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0: No bootstrap required, opened a new log
I20260810 07:28:35.299624 14207 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc89b5beac5c4e729e14089414cc44f0" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40355 } }
I20260810 07:28:35.299765 14207 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.299788 14207 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc89b5beac5c4e729e14089414cc44f0, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.299974 14207 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc89b5beac5c4e729e14089414cc44f0" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40355 } }
I20260810 07:28:35.300040 14207 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:35.300061 14207 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:35.300134 14207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.300732 14207 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc89b5beac5c4e729e14089414cc44f0" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40355 } }
I20260810 07:28:35.300861 14207 leader_election.cc:302] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc89b5beac5c4e729e14089414cc44f0; no voters:
I20260810 07:28:35.300890 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 14128
I20260810 07:28:35.300998 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:35.301070 14207 leader_election.cc:288] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:35.301198 14211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:35.301270 24819 external_mini_cluster.cc:1428] Setting key db05b183eddc1bfab1c2e8e9c8ca4744
I20260810 07:28:35.301366 14207 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:35.301378 14211 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [term 1 LEADER]: Becoming Leader. State: Replica: cc89b5beac5c4e729e14089414cc44f0, State: Running, Role: LEADER
I20260810 07:28:35.301626 14211 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc89b5beac5c4e729e14089414cc44f0" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40355 } }
I20260810 07:28:35.302014 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc89b5beac5c4e729e14089414cc44f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc89b5beac5c4e729e14089414cc44f0" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40355 } } }
I20260810 07:28:35.302096 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc89b5beac5c4e729e14089414cc44f0. Latest consensus state: current_term: 1 leader_uuid: "cc89b5beac5c4e729e14089414cc44f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc89b5beac5c4e729e14089414cc44f0" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40355 } } }
I20260810 07:28:35.302223 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:35.302191 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:35.302461 14224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:35.303267 14224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:35.304900 14224 catalog_manager.cc:1347] Generated new cluster ID: 16d5e05a012243519aafc5b4f2824702
I20260810 07:28:35.304953 14224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:35.308097 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.301776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.24.60.254:40355)
0810 07:28:35.302006 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.302016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.302624 (+ 608us) server_negotiation.cc:407] Connection header received
0810 07:28:35.302854 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.302861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.302939 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.303059 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.303476 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.304632 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.305793 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.306448 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.306622 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.307333 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.307337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.307337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:35.307559 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.307603 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:35.307619 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.307696 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.307769 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.307857 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260810 07:28:35.320617 14224 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:35.321198 14224 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:35.327351 14224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0: Generated new TSK 0
I20260810 07:28:35.327582 14224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:35.334569 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:40355
--builtin_ntp_servers=127.24.60.212:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:35.452651 14235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:35.452893 14235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:35.452919 14235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:35.454309 14235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:35.454360 14235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:35.454388 14235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:35.455787 14235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:40355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:35.456067 14235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:35.456418 14235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:35.462618 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.462816 14242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.462594 14243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:35.462973 14235 server_base.cc:1034] running on GCE node
I20260810 07:28:35.463217 14235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:35.463475 14235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:35.464643 14235 hybrid_clock.cc:648] HybridClock initialized: now 1786346915464622 us; error 38 us; skew 500 ppm
I20260810 07:28:35.466123 14235 webserver.cc:460] Webserver started at http://127.24.60.193:35329/ using document root <none> and password file <none>
I20260810 07:28:35.466429 14235 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:35.466526 14235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:35.466686 14235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:35.467942 14235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "bf2734b0446448458654c229bae2041e"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "0445dd7690c3d91e3b3bb0232ec7476b"
server_key_iv: "52c4509827efc7ab4fe1ec0cf7434fb4"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.468319 14235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2734b0446448458654c229bae2041e"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "0445dd7690c3d91e3b3bb0232ec7476b"
server_key_iv: "52c4509827efc7ab4fe1ec0cf7434fb4"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.470031 14235 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 07:28:35.470939 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.471164 14235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:35.471266 14235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "bf2734b0446448458654c229bae2041e"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "0445dd7690c3d91e3b3bb0232ec7476b"
server_key_iv: "52c4509827efc7ab4fe1ec0cf7434fb4"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.471357 14235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:35.497198 14235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:35.497593 14235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:35.497761 14235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:35.498021 14235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:35.498374 14235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:35.498442 14235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.498490 14235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:35.498515 14235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.505004 14235 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:36897
I20260810 07:28:35.505046 14389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:36897 every 8 connection(s)
I20260810 07:28:35.505491 14235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:35.511013 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.506422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:37279 (local address 127.24.60.254:40355)
0810 07:28:35.506486 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.506488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.506784 (+ 296us) server_negotiation.cc:407] Connection header received
0810 07:28:35.507289 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.507290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.507306 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.507362 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.508084 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.508583 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.509607 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.509797 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.509848 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.510797 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.510797 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.510798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:35.510803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.510814 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:35.510817 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.510856 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.510921 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.510949 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:28:35.511222 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.506411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:40355 (local address 127.24.60.193:37279)
0810 07:28:35.506646 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.506686 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.507196 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.507416 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.507419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.507434 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:35.507983 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.507987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.508675 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.508676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.509529 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.509531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.509875 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.509876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.509993 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.510658 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:35.510659 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:35.510689 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:35.510713 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:35.510857 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:35.510859 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:35.510862 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.510911 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.510998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":128,"threads_started":1}
I20260810 07:28:35.511776 14390 heartbeater.cc:347] Connected to a master server at 127.24.60.254:40355
I20260810 07:28:35.511899 14390 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:35.512109 14390 heartbeater.cc:511] Master 127.24.60.254:40355 requested a full tablet report, sending...
I20260810 07:28:35.512142 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 14235
I20260810 07:28:35.512252 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:35.512490 24819 external_mini_cluster.cc:1428] Setting key 2e6ff75cbae9f33411119a0904ed6d41
I20260810 07:28:35.512712 14167 ts_manager.cc:194] Registered new tserver with Master: bf2734b0446448458654c229bae2041e (127.24.60.193:36897)
I20260810 07:28:35.513674 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:40355
--builtin_ntp_servers=127.24.60.212:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:28:35.513666 14167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.193:37279
W20260810 07:28:35.600538 14393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:35.600781 14393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:35.600806 14393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:35.602075 14393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:35.602123 14393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:35.602149 14393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:35.603549 14393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:40355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:35.603755 14393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:35.604032 14393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:35.609284 14406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.609259 14402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:35.609292 14393 server_base.cc:1034] running on GCE node
W20260810 07:28:35.609258 14403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:35.609757 14393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:35.610092 14393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:35.611275 14393 hybrid_clock.cc:648] HybridClock initialized: now 1786346915611256 us; error 38 us; skew 500 ppm
I20260810 07:28:35.612648 14393 webserver.cc:460] Webserver started at http://127.24.60.194:40443/ using document root <none> and password file <none>
I20260810 07:28:35.612921 14393 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:35.612998 14393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:35.613198 14393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:35.614352 14393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "908b83a35de74525a5890431c1a9045c"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "9d56ac11da430b9b00fe228ca983087e"
server_key_iv: "2306d27113dc34a66fb3c0ea78a71525"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.614742 14393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "908b83a35de74525a5890431c1a9045c"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "9d56ac11da430b9b00fe228ca983087e"
server_key_iv: "2306d27113dc34a66fb3c0ea78a71525"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.616194 14393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:35.616958 14412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.617184 14393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 07:28:35.617301 14393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "908b83a35de74525a5890431c1a9045c"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "9d56ac11da430b9b00fe228ca983087e"
server_key_iv: "2306d27113dc34a66fb3c0ea78a71525"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.617391 14393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:35.625536 14393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:35.625854 14393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:35.626016 14393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:35.626248 14393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:35.626677 14393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:35.626739 14393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.626784 14393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:35.626813 14393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.632982 14393 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:43671
I20260810 07:28:35.633023 14544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:43671 every 8 connection(s)
I20260810 07:28:35.633507 14393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:35.638919 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.634312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:59079 (local address 127.24.60.254:40355)
0810 07:28:35.634377 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.634379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.634638 (+ 259us) server_negotiation.cc:407] Connection header received
0810 07:28:35.635124 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.635125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.635135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.635191 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.635990 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.636479 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.637559 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.637770 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.637846 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.638650 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.638651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.638652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:35.638657 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.638671 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:35.638675 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.638729 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.638825 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.638854 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:28:35.639092 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.634300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:40355 (local address 127.24.60.194:59079)
0810 07:28:35.634511 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.634546 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.635038 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.635299 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.635305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.635315 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:35.635906 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.635912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.636564 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.636564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.637450 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.637453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.637858 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.637859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.637975 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.638513 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:35.638514 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:35.638542 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:35.638565 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:35.638738 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:35.638742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:35.638744 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.638796 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.638876 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260810 07:28:35.639355 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 14393
I20260810 07:28:35.639519 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:35.639731 14545 heartbeater.cc:347] Connected to a master server at 127.24.60.254:40355
I20260810 07:28:35.639781 24819 external_mini_cluster.cc:1428] Setting key b77c863bf06921b12ad408a683a92254
I20260810 07:28:35.639851 14545 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:35.640057 14545 heartbeater.cc:511] Master 127.24.60.254:40355 requested a full tablet report, sending...
I20260810 07:28:35.640489 14166 ts_manager.cc:194] Registered new tserver with Master: 908b83a35de74525a5890431c1a9045c (127.24.60.194:43671)
I20260810 07:28:35.640915 14166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.194:59079
I20260810 07:28:35.640946 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:40355
--builtin_ntp_servers=127.24.60.212:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:35.725380 14550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:35.725601 14550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:35.725625 14550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:35.726856 14550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:35.726907 14550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:35.726940 14550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:35.728358 14550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:40355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:35.728619 14550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:35.728945 14550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:35.734280 14561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.734280 14565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:35.734308 14563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:35.734510 14550 server_base.cc:1034] running on GCE node
I20260810 07:28:35.734793 14550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:35.735060 14550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:35.736251 14550 hybrid_clock.cc:648] HybridClock initialized: now 1786346915736246 us; error 37 us; skew 500 ppm
I20260810 07:28:35.737574 14550 webserver.cc:460] Webserver started at http://127.24.60.195:45221/ using document root <none> and password file <none>
I20260810 07:28:35.737857 14550 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:35.737932 14550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:35.738096 14550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:35.739279 14550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "9986f8c1a5884908950dce5fee6a63d1"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "1eb023c2e75391d0fefd95ef20319041"
server_key_iv: "14964accd43dd284069c67335466a57b"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.739655 14550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "9986f8c1a5884908950dce5fee6a63d1"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "1eb023c2e75391d0fefd95ef20319041"
server_key_iv: "14964accd43dd284069c67335466a57b"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.741065 14550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:35.741863 14573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.742049 14550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 07:28:35.742161 14550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "9986f8c1a5884908950dce5fee6a63d1"
format_stamp: "Formatted at 2026-08-10 07:28:35 on dist-test-slave-vxj2"
server_key: "1eb023c2e75391d0fefd95ef20319041"
server_key_iv: "14964accd43dd284069c67335466a57b"
server_key_version: "encryptionkey@0"
I20260810 07:28:35.742237 14550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:35.757856 14550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:35.758229 14550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:35.758399 14550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:35.758664 14550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:35.759021 14550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:35.759078 14550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.759119 14550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:35.759161 14550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:35.765106 14550 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:33465
I20260810 07:28:35.765228 14703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:33465 every 8 connection(s)
I20260810 07:28:35.765571 14550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:35.766359 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 14550
I20260810 07:28:35.766489 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:35.766747 24819 external_mini_cluster.cc:1428] Setting key 349a09e8cd79bbfad4d7bfc50a1bba6b
I20260810 07:28:35.771368 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.766442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:51751 (local address 127.24.60.254:40355)
0810 07:28:35.766528 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.766530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.766833 (+ 303us) server_negotiation.cc:407] Connection header received
0810 07:28:35.767551 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.767552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.767566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.767616 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.768414 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.768928 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.769938 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.770130 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.770191 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.771112 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.771113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.771114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:35.771119 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.771136 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:35.771139 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.771183 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.771276 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.771304 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:28:35.771631 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.766460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:40355 (local address 127.24.60.195:51751)
0810 07:28:35.766692 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.766732 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.767426 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.767688 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.767691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.767716 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:35.768322 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.768328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.769022 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.769023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.769853 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.769855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.770291 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.770293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.770430 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.770966 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:35.770966 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:35.770998 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:35.771023 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:35.771199 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:35.771203 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:35.771206 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.771262 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.771350 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260810 07:28:35.772208 14705 heartbeater.cc:347] Connected to a master server at 127.24.60.254:40355
I20260810 07:28:35.772325 14705 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:35.772522 14705 heartbeater.cc:511] Master 127.24.60.254:40355 requested a full tablet report, sending...
I20260810 07:28:35.772922 14166 ts_manager.cc:194] Registered new tserver with Master: 9986f8c1a5884908950dce5fee6a63d1 (127.24.60.195:33465)
I20260810 07:28:35.773351 14166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.195:51751
I20260810 07:28:35.778872 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:35.783470 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.780488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.24.60.254:40355)
0810 07:28:35.780546 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.780548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.780725 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:28:35.780756 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.780757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.780769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.780819 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:35.780995 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.781577 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.782329 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.782527 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.782579 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.783219 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:35.783220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:35.783220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:35.783225 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:35.783239 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:35.783243 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:35.783324 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.783386 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.783397 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:28:35.785399 14166 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:35.787038 14166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:35.794106 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.789361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:43671 (local address 127.0.0.1:49104)
0810 07:28:35.789797 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.789808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.789863 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.790175 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.790196 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.790199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.790286 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.790288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.791649 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.791650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.792698 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.792699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.793690 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.793691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.793817 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.793839 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.793959 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.794016 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":8,"thread_start_us":56,"threads_started":1}
I20260810 07:28:35.794127 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.789450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.24.60.194:43671)
0810 07:28:35.789653 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.789659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.789953 (+ 294us) server_negotiation.cc:407] Connection header received
0810 07:28:35.790005 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.790007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.790024 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.790098 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.790395 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.791557 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.792840 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.793593 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.793680 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.793879 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.793980 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.794027 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260810 07:28:35.794144 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.789430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.24.60.195:33465)
0810 07:28:35.789702 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.789708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.789847 (+ 139us) server_negotiation.cc:407] Connection header received
0810 07:28:35.789958 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.789961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.789979 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.790063 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.790489 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.791209 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.792687 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.793593 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.793691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.793787 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.794015 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.794057 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20260810 07:28:35.794407 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.789356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:33465 (local address 127.0.0.1:43730)
0810 07:28:35.789715 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.789780 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.789863 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.790322 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.790323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.790326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.790374 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.790376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.791305 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.791306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.792566 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.792569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.793827 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.793828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.793899 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.793903 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.794299 (+ 396us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.794354 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":98,"threads_started":1}
I20260810 07:28:35.794842 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.789587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36897 (local address 127.0.0.1:59748)
0810 07:28:35.789980 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.789989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.790039 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.790216 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.790217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.790218 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.790286 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.790288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.792779 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.792779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.793510 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.793513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.794574 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.794575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.794638 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.794644 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.794704 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.794764 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260810 07:28:35.795167 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.789676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.24.60.193:36897)
0810 07:28:35.789828 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.789833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.790040 (+ 207us) server_negotiation.cc:407] Connection header received
0810 07:28:35.790110 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.790112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.790128 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.790211 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.790382 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.791558 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.793601 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.794483 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.794550 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.794632 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.795033 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.795083 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260810 07:28:35.795421 14464 tablet_service.cc:1467] Processing CreateTablet for tablet fecf37c40d8948da8fde48dd2dc10392 (DEFAULT_TABLE table=test-table [id=bd53129fab68408b8dab83a2acc2f4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:35.795411 14632 tablet_service.cc:1467] Processing CreateTablet for tablet fecf37c40d8948da8fde48dd2dc10392 (DEFAULT_TABLE table=test-table [id=bd53129fab68408b8dab83a2acc2f4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:35.795845 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf37c40d8948da8fde48dd2dc10392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.795848 14632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf37c40d8948da8fde48dd2dc10392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.795805 14309 tablet_service.cc:1467] Processing CreateTablet for tablet fecf37c40d8948da8fde48dd2dc10392 (DEFAULT_TABLE table=test-table [id=bd53129fab68408b8dab83a2acc2f4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:35.796139 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf37c40d8948da8fde48dd2dc10392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:35.798866 14720 tablet_bootstrap.cc:492] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Bootstrap starting.
I20260810 07:28:35.799578 14722 tablet_bootstrap.cc:492] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: Bootstrap starting.
I20260810 07:28:35.799649 14720 tablet_bootstrap.cc:654] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.799971 14720 log.cc:824] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.800209 14722 tablet_bootstrap.cc:654] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.800546 14722 log.cc:824] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.800889 14721 tablet_bootstrap.cc:492] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Bootstrap starting.
I20260810 07:28:35.800993 14720 tablet_bootstrap.cc:492] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: No bootstrap required, opened a new log
I20260810 07:28:35.801075 14720 ts_tablet_manager.cc:1397] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:28:35.801373 14722 tablet_bootstrap.cc:492] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: No bootstrap required, opened a new log
I20260810 07:28:35.801455 14722 ts_tablet_manager.cc:1397] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:28:35.801589 14721 tablet_bootstrap.cc:654] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:35.801932 14721 log.cc:824] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:35.803792 14722 raft_consensus.cc:348] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.803792 14720 raft_consensus.cc:348] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.803938 14722 raft_consensus.cc:374] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.803962 14722 raft_consensus.cc:729] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2734b0446448458654c229bae2041e, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.803964 14720 raft_consensus.cc:374] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.804009 14720 raft_consensus.cc:729] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908b83a35de74525a5890431c1a9045c, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.804095 14722 consensus_queue.cc:260] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.804215 14720 consensus_queue.cc:260] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.804378 14722 ts_tablet_manager.cc:1428] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:35.804428 14390 heartbeater.cc:503] Master 127.24.60.254:40355 was elected leader, sending a full tablet report...
I20260810 07:28:35.804543 14720 ts_tablet_manager.cc:1428] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 07:28:35.804621 14545 heartbeater.cc:503] Master 127.24.60.254:40355 was elected leader, sending a full tablet report...
I20260810 07:28:35.805290 14721 tablet_bootstrap.cc:492] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: No bootstrap required, opened a new log
I20260810 07:28:35.805371 14721 ts_tablet_manager.cc:1397] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 07:28:35.807591 14721 raft_consensus.cc:348] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.807736 14721 raft_consensus.cc:374] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:35.807765 14721 raft_consensus.cc:729] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9986f8c1a5884908950dce5fee6a63d1, State: Initialized, Role: FOLLOWER
I20260810 07:28:35.807928 14721 consensus_queue.cc:260] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.808233 14721 ts_tablet_manager.cc:1428] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:35.808275 14705 heartbeater.cc:503] Master 127.24.60.254:40355 was elected leader, sending a full tablet report...
W20260810 07:28:35.884096 14547 tablet.cc:2369] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:35.984054 14727 raft_consensus.cc:482] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:35.984198 14727 raft_consensus.cc:504] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.984755 14727 leader_election.cc:288] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9986f8c1a5884908950dce5fee6a63d1 (127.24.60.195:33465), bf2734b0446448458654c229bae2041e (127.24.60.193:36897)
I20260810 07:28:35.988986 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.985120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:41319 (local address 127.24.60.193:36897)
0810 07:28:35.985216 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.985219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.985319 (+ 100us) server_negotiation.cc:407] Connection header received
0810 07:28:35.985358 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.985359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.985372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.985433 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.985664 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.986448 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.987997 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.988576 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.988651 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.988660 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.988865 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.988910 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 07:28:35.988986 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.985126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:40523 (local address 127.24.60.195:33465)
0810 07:28:35.985220 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:35.985222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:35.985233 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:28:35.985279 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:35.985280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:35.985308 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:35.985380 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:35.985658 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.986465 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.988003 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.988561 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.988651 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.988660 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:35.988865 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 07:28:35.988910 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 07:28:35.989032 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.984967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:33465 (local address 127.24.60.194:40523)
0810 07:28:35.985070 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.985086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.985141 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.985439 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.985440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.985441 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.985521 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.985523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.986559 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.986560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.987868 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.987873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.988672 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.988673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.988771 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.988775 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.988837 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.988896 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":35}
I20260810 07:28:35.989048 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:35.985025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36897 (local address 127.24.60.194:41319)
0810 07:28:35.985235 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:35.985244 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:35.985300 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:35.985460 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:35.985461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:35.985462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:35.985543 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.985544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.986559 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.986560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.987869 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:35.987873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:35.988672 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:35.988673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:35.988772 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:35.988775 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:35.988837 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 07:28:35.988896 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":117,"thread_start_us":111,"threads_started":1}
I20260810 07:28:35.989459 14656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf37c40d8948da8fde48dd2dc10392" candidate_uuid: "908b83a35de74525a5890431c1a9045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9986f8c1a5884908950dce5fee6a63d1" is_pre_election: true
I20260810 07:28:35.989661 14656 raft_consensus.cc:2454] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908b83a35de74525a5890431c1a9045c in term 0.
I20260810 07:28:35.990008 14415 leader_election.cc:302] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908b83a35de74525a5890431c1a9045c, 9986f8c1a5884908950dce5fee6a63d1; no voters:
I20260810 07:28:35.990024 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf37c40d8948da8fde48dd2dc10392" candidate_uuid: "908b83a35de74525a5890431c1a9045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2734b0446448458654c229bae2041e" is_pre_election: true
I20260810 07:28:35.990172 14337 raft_consensus.cc:2454] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908b83a35de74525a5890431c1a9045c in term 0.
I20260810 07:28:35.990195 14727 raft_consensus.cc:2781] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:35.990255 14727 raft_consensus.cc:482] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:35.990290 14727 raft_consensus.cc:3037] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.991050 14727 raft_consensus.cc:504] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.991223 14727 leader_election.cc:288] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [CANDIDATE]: Term 1 election: Requested vote from peers 9986f8c1a5884908950dce5fee6a63d1 (127.24.60.195:33465), bf2734b0446448458654c229bae2041e (127.24.60.193:36897)
I20260810 07:28:35.991400 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf37c40d8948da8fde48dd2dc10392" candidate_uuid: "908b83a35de74525a5890431c1a9045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2734b0446448458654c229bae2041e"
I20260810 07:28:35.991437 14656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf37c40d8948da8fde48dd2dc10392" candidate_uuid: "908b83a35de74525a5890431c1a9045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9986f8c1a5884908950dce5fee6a63d1"
I20260810 07:28:35.991496 14337 raft_consensus.cc:3037] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.991508 14656 raft_consensus.cc:3037] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:35.992468 14337 raft_consensus.cc:2454] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908b83a35de74525a5890431c1a9045c in term 1.
I20260810 07:28:35.992468 14656 raft_consensus.cc:2454] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908b83a35de74525a5890431c1a9045c in term 1.
I20260810 07:28:35.992726 14416 leader_election.cc:302] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908b83a35de74525a5890431c1a9045c, bf2734b0446448458654c229bae2041e; no voters:
I20260810 07:28:35.992923 14727 raft_consensus.cc:2781] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:35.993198 14727 raft_consensus.cc:686] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 1 LEADER]: Becoming Leader. State: Replica: 908b83a35de74525a5890431c1a9045c, State: Running, Role: LEADER
I20260810 07:28:35.993319 14727 consensus_queue.cc:237] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } }
I20260810 07:28:35.994242 14166 catalog_manager.cc:5495] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c reported cstate change: term changed from 0 to 1, leader changed from <none> to 908b83a35de74525a5890431c1a9045c (127.24.60.194). New cstate: current_term: 1 leader_uuid: "908b83a35de74525a5890431c1a9045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "908b83a35de74525a5890431c1a9045c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 } health_report { overall_health: UNKNOWN } } }
W20260810 07:28:36.006330 14391 tablet.cc:2369] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:36.016187 14706 tablet.cc:2369] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:36.026527 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:36.023951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.24.60.194:43671)
0810 07:28:36.024033 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:36.024035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:36.024042 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:28:36.024081 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:36.024081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:36.024102 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:36.024185 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:36.024338 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:36.024814 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:36.025690 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:36.025897 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:36.025977 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:36.026136 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:36.026227 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:36.026338 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:36.026399 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 07:28:36.026408 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:28:36.028529 14337 raft_consensus.cc:1261] T fecf37c40d8948da8fde48dd2dc10392 P bf2734b0446448458654c229bae2041e [term 1 FOLLOWER]: Refusing update from remote peer 908b83a35de74525a5890431c1a9045c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:36.028538 14656 raft_consensus.cc:1261] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 1 FOLLOWER]: Refusing update from remote peer 908b83a35de74525a5890431c1a9045c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:36.028931 14727 consensus_queue.cc:1035] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2734b0446448458654c229bae2041e" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 36897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:36.029135 14732 consensus_queue.cc:1035] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9986f8c1a5884908950dce5fee6a63d1" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 33465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:36.031490 14735 mvcc.cc:205] Tried to move back new op lower bound from 7316876968049856512 to 7316876967909216256. Current Snapshot: MvccSnapshot[applied={T|T < 7316876968049856512}]
I20260810 07:28:36.034147 14167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:36.034264 14167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:36.035125 14167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc89b5beac5c4e729e14089414cc44f0: Sending DeleteTablet for 3 replicas of tablet fecf37c40d8948da8fde48dd2dc10392
I20260810 07:28:36.035475 14464 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf37c40d8948da8fde48dd2dc10392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:36 UTC) from {username='slave'} at 127.0.0.1:49104
I20260810 07:28:36.035573 14309 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf37c40d8948da8fde48dd2dc10392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:36 UTC) from {username='slave'} at 127.0.0.1:59748
I20260810 07:28:36.035578 14632 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf37c40d8948da8fde48dd2dc10392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:36 UTC) from {username='slave'} at 127.0.0.1:43730
I20260810 07:28:36.035704 14746 tablet_replica.cc:330] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: stopping tablet replica
I20260810 07:28:36.035934 14746 raft_consensus.cc:2229] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:28:36.036072 14746 raft_consensus.cc:2258] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:36.036120 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 14235
I20260810 07:28:36.037832 14746 ts_tablet_manager.cc:1905] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:36.039688 14746 ts_tablet_manager.cc:1918] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:36.039773 14746 log.cc:1196] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/fecf37c40d8948da8fde48dd2dc10392
I20260810 07:28:36.040050 14746 ts_tablet_manager.cc:1939] T fecf37c40d8948da8fde48dd2dc10392 P 908b83a35de74525a5890431c1a9045c: Deleting consensus metadata
I20260810 07:28:36.042615 14151 catalog_manager.cc:4841] TS 908b83a35de74525a5890431c1a9045c (127.24.60.194:43671): tablet fecf37c40d8948da8fde48dd2dc10392 (table test-table [id=bd53129fab68408b8dab83a2acc2f4b3]) successfully deleted
I20260810 07:28:36.042881 14748 tablet_replica.cc:330] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: stopping tablet replica
I20260810 07:28:36.043361 14748 raft_consensus.cc:2229] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:36.043524 14748 raft_consensus.cc:2258] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:36.044040 14748 ts_tablet_manager.cc:1905] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:36.046224 14748 ts_tablet_manager.cc:1918] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:36.046314 14748 log.cc:1196] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/fecf37c40d8948da8fde48dd2dc10392
I20260810 07:28:36.046618 14748 ts_tablet_manager.cc:1939] T fecf37c40d8948da8fde48dd2dc10392 P 9986f8c1a5884908950dce5fee6a63d1: Deleting consensus metadata
I20260810 07:28:36.047254 14150 catalog_manager.cc:4841] TS 9986f8c1a5884908950dce5fee6a63d1 (127.24.60.195:33465): tablet fecf37c40d8948da8fde48dd2dc10392 (table test-table [id=bd53129fab68408b8dab83a2acc2f4b3]) successfully deleted
W20260810 07:28:36.048389 14151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:36897 (error 108)
I20260810 07:28:36.048666 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 14393
I20260810 07:28:36.048894 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:36.048753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:36897 (local address 127.0.0.1:59764)
0810 07:28:36.048802 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:36.048851 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 07:28:36.052840 14151 catalog_manager.cc:4584] TS bf2734b0446448458654c229bae2041e (127.24.60.193:36897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fecf37c40d8948da8fde48dd2dc10392: Network error: Client connection negotiation failed: client connection to 127.24.60.193:36897: connect: Connection refused (error 111)
I20260810 07:28:36.059795 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 14550
I20260810 07:28:36.068017 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 14128
2026-08-10T07:28:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (923 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:36 dist-test-slave-vxj2 krb5kdc[14756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:36 dist-test-slave-vxj2 krb5kdc[14756](info): set up 2 sockets
Aug 10 07:28:36 dist-test-slave-vxj2 krb5kdc[14756](info): commencing operation
krb5kdc: starting...
W20260810 07:28:37.195914 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:28:37.322300 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44883
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:44883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:37.411013 14772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:37.411239 14772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:37.411263 14772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:37.412477 14772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:37.412524 14772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:37.412539 14772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:37.412551 14772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:37.412563 14772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:37.414206 14772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:44883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:37.414472 14772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:37.414746 14772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:37.419833 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:37.419875 14780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:37.419875 14782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:37.420049 14772 server_base.cc:1034] running on GCE node
I20260810 07:28:37.420423 14772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:37.420811 14772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:37.422020 14772 hybrid_clock.cc:648] HybridClock initialized: now 1786346917421994 us; error 38 us; skew 500 ppm
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:37.424698 14772 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:28:37.425233 14772 webserver.cc:460] Webserver started at http://127.24.60.254:34555/ using document root <none> and password file <none>
I20260810 07:28:37.425503 14772 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:37.425575 14772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:37.425729 14772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:37.426844 14772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "bf46352f9b6e4556855ec65787ba90b3"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "2c2139ddbc45f46a54ad9c37e8a1e33c"
server_key_iv: "8cf415d56aa754b77eeffffe86df68b0"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.427234 14772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "bf46352f9b6e4556855ec65787ba90b3"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "2c2139ddbc45f46a54ad9c37e8a1e33c"
server_key_iv: "8cf415d56aa754b77eeffffe86df68b0"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.428627 14772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:37.429447 14791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.429646 14772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:37.429759 14772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "bf46352f9b6e4556855ec65787ba90b3"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "2c2139ddbc45f46a54ad9c37e8a1e33c"
server_key_iv: "8cf415d56aa754b77eeffffe86df68b0"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.429828 14772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:37.453624 14772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:37.455802 14772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:37.455981 14772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:37.460186 14772 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:44883
I20260810 07:28:37.460249 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:44883 every 8 connection(s)
I20260810 07:28:37.460649 14772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:37.461642 14850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:37.463953 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3: Bootstrap starting.
I20260810 07:28:37.464560 14850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:37.464861 14850 log.cc:824] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:37.465694 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3: No bootstrap required, opened a new log
I20260810 07:28:37.467373 14850 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf46352f9b6e4556855ec65787ba90b3" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44883 } }
I20260810 07:28:37.467527 14850 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:37.467550 14850 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf46352f9b6e4556855ec65787ba90b3, State: Initialized, Role: FOLLOWER
I20260810 07:28:37.467710 14850 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf46352f9b6e4556855ec65787ba90b3" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44883 } }
I20260810 07:28:37.467789 14850 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:37.467811 14850 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:37.467882 14850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:37.468261 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 14772
I20260810 07:28:37.468362 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:37.468586 24819 external_mini_cluster.cc:1428] Setting key 060b13f7966fde407e87b61dc28bc916
I20260810 07:28:37.468597 14850 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf46352f9b6e4556855ec65787ba90b3" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44883 } }
I20260810 07:28:37.468735 14850 leader_election.cc:302] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf46352f9b6e4556855ec65787ba90b3; no voters:
I20260810 07:28:37.468972 14850 leader_election.cc:288] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:37.469031 14854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:37.469373 14854 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [term 1 LEADER]: Becoming Leader. State: Replica: bf46352f9b6e4556855ec65787ba90b3, State: Running, Role: LEADER
I20260810 07:28:37.469511 14850 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:37.469537 14854 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf46352f9b6e4556855ec65787ba90b3" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44883 } }
I20260810 07:28:37.469892 14854 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf46352f9b6e4556855ec65787ba90b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf46352f9b6e4556855ec65787ba90b3" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44883 } } }
I20260810 07:28:37.469980 14854 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:37.470134 14854 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf46352f9b6e4556855ec65787ba90b3. Latest consensus state: current_term: 1 leader_uuid: "bf46352f9b6e4556855ec65787ba90b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf46352f9b6e4556855ec65787ba90b3" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44883 } } }
I20260810 07:28:37.470199 14854 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:37.470294 14866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:37.470877 14866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:37.477806 14866 catalog_manager.cc:1347] Generated new cluster ID: e3aa5a13e5bd48dfbaa5a818c88c49c4
I20260810 07:28:37.477872 14866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:37.480604 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:37.469137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.24.60.254:44883)
0810 07:28:37.470889 (+ 1752us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:37.470901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:37.470954 (+ 53us) server_negotiation.cc:407] Connection header received
0810 07:28:37.471098 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:37.471106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:37.471171 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:37.471338 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:37.472102 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.473057 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.473909 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.474343 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.474477 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:37.476950 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:37.476955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:37.476955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:37.476970 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:37.478919 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:37.479314 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:37.479318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:37.479318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:37.479431 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:37.479658 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:37.479659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:37.479659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:37.479916 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:37.480097 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:37.480197 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 07:28:37.480340 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1664,"thread_start_us":175,"threads_started":1}
I20260810 07:28:37.496241 14866 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:37.496803 14866 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:37.501574 14866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3: Generated new TSK 0
I20260810 07:28:37.501780 14866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.60.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
Entry for principal kudu/127.24.60.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab.
I20260810 07:28:37.532626 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44883
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:37.617818 14884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:37.618033 14884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:37.618062 14884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:37.619303 14884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:37.619349 14884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:37.619378 14884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:37.620923 14884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.193.keytab
--principal=kudu/127.24.60.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:37.621204 14884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:37.621470 14884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:37.626736 14891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:37.626724 14890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:37.626727 14893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:37.627077 14884 server_base.cc:1034] running on GCE node
I20260810 07:28:37.627377 14884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:37.627637 14884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:37.628816 14884 hybrid_clock.cc:648] HybridClock initialized: now 1786346917628793 us; error 36 us; skew 500 ppm
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:37.631605 14884 init.cc:377] Logged in from keytab as kudu/127.24.60.193@KRBTEST.COM (short username kudu)
I20260810 07:28:37.632081 14884 webserver.cc:460] Webserver started at http://127.24.60.193:38835/ using document root <none> and password file <none>
I20260810 07:28:37.632313 14884 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:37.632359 14884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:37.632546 14884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:37.633755 14884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "430b6f58602e41a0ac179e1ca5f5aa33"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "18ac3e02fef25729c04738a7e3c32aab"
server_key_iv: "b9f55e4ca4e5796e5f2b03e381781a51"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.634125 14884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "430b6f58602e41a0ac179e1ca5f5aa33"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "18ac3e02fef25729c04738a7e3c32aab"
server_key_iv: "b9f55e4ca4e5796e5f2b03e381781a51"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.635529 14884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:28:37.636354 14902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.636562 14884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:37.636672 14884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "430b6f58602e41a0ac179e1ca5f5aa33"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "18ac3e02fef25729c04738a7e3c32aab"
server_key_iv: "b9f55e4ca4e5796e5f2b03e381781a51"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.636758 14884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:37.660609 14884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:37.662984 14884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:37.663162 14884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:37.663429 14884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:37.663772 14884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:37.663829 14884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.663865 14884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:37.663915 14884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.669942 14884 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:33577
I20260810 07:28:37.670023 15032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:33577 every 8 connection(s)
I20260810 07:28:37.670431 14884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.193@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:37.678732 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 14884
I20260810 07:28:37.678877 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:37.679108 24819 external_mini_cluster.cc:1428] Setting key 32861428d4d87d03ea6d128dc9e90081
I20260810 07:28:37.679855 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:37.671256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:35873 (local address 127.24.60.254:44883)
0810 07:28:37.671331 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:37.671334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:37.671584 (+ 250us) server_negotiation.cc:407] Connection header received
0810 07:28:37.672075 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:37.672077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:37.672088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:37.672152 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:37.673118 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.673651 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.674664 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.674852 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.674903 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:37.677059 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:37.677061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:37.677061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:37.677070 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:37.678417 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:37.678833 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:37.678834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:37.678835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:37.678896 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:37.679177 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:37.679178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:37.679179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:37.679305 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:37.679381 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:37.679673 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 07:28:37.679767 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:28:37.680045 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:37.671237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44883 (local address 127.24.60.193:35873)
0810 07:28:37.671440 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:37.671493 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:37.671992 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:37.672206 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:37.672210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:37.672818 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:37.673028 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:37.673032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.673738 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.673739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:37.674559 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:37.674578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.674929 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.674929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:37.675033 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:37.675518 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:37.675526 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:37.676955 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:37.678510 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:37.678518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:37.678520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:37.678753 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:37.678990 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:37.678991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:37.678991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:37.679101 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:37.679382 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:37.679391 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:37.679532 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 07:28:37.679634 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:28:37.679796 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
I20260810 07:28:37.680527 15033 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44883
I20260810 07:28:37.680644 15033 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:37.680838 15033 heartbeater.cc:511] Master 127.24.60.254:44883 requested a full tablet report, sending...
I20260810 07:28:37.681393 14809 ts_manager.cc:194] Registered new tserver with Master: 430b6f58602e41a0ac179e1ca5f5aa33 (127.24.60.193:33577)
I20260810 07:28:37.682022 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.193@KRBTEST.COM'} at 127.24.60.193:35873
WARNING: no policy specified for kudu/127.24.60.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
Entry for principal kudu/127.24.60.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab.
I20260810 07:28:37.708312 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44883
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:37.804702 15038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:37.804947 15038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:37.804980 15038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:37.806226 15038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:37.806282 15038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:37.806313 15038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:37.807776 15038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.194.keytab
--principal=kudu/127.24.60.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:37.807982 15038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:37.808176 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:37.813505 15044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:37.813681 15038 server_base.cc:1034] running on GCE node
W20260810 07:28:37.813516 15043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:37.813756 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:37.814015 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:37.814241 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:37.815382 15038 hybrid_clock.cc:648] HybridClock initialized: now 1786346917815358 us; error 41 us; skew 500 ppm
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:37.818105 15038 init.cc:377] Logged in from keytab as kudu/127.24.60.194@KRBTEST.COM (short username kudu)
I20260810 07:28:37.818526 15038 webserver.cc:460] Webserver started at http://127.24.60.194:36463/ using document root <none> and password file <none>
I20260810 07:28:37.818718 15038 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:37.818759 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:37.818861 15038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:37.819919 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "9dd9cf32fac14889be1aafcca8d69417"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "d83418abd09be58131bf7baacc87a801"
server_key_iv: "ec1ee606f1a91203e432c11d91b9b7a0"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.820231 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "9dd9cf32fac14889be1aafcca8d69417"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "d83418abd09be58131bf7baacc87a801"
server_key_iv: "ec1ee606f1a91203e432c11d91b9b7a0"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.821662 15038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 07:28:37.822551 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.822754 15038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:37.822844 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "9dd9cf32fac14889be1aafcca8d69417"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "d83418abd09be58131bf7baacc87a801"
server_key_iv: "ec1ee606f1a91203e432c11d91b9b7a0"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.822970 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:37.833637 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:37.835868 15038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:37.836021 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:37.836218 15038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:37.836476 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:37.836508 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.836530 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:37.836544 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.842666 15038 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:41065
I20260810 07:28:37.842798 15189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:41065 every 8 connection(s)
I20260810 07:28:37.843102 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:37.844271 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 15038
I20260810 07:28:37.844425 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:37.844628 24819 external_mini_cluster.cc:1428] Setting key f21e3281fab1cfab1b955180e6ad822b
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.194@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.195@KRBTEST.COM; defaulting to no policy
I20260810 07:28:37.854727 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:37.843955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:33619 (local address 127.24.60.254:44883)
0810 07:28:37.844066 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:37.844069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:37.844239 (+ 170us) server_negotiation.cc:407] Connection header received
0810 07:28:37.844956 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:37.844958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:37.844971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:37.845039 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:37.846205 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.846993 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.848591 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.848779 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.848834 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:37.851648 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:37.851650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:37.851650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:37.851659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:37.853201 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:37.853725 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:37.853726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:37.853726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:37.853781 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:37.854057 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:37.854057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:37.854057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:37.854197 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:37.854275 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:37.854583 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 07:28:37.854656 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:37.854939 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:37.843878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44883 (local address 127.24.60.194:33619)
0810 07:28:37.844116 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:37.844154 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:37.844812 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:37.845114 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:37.845119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:37.845838 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:37.846094 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:37.846099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.847120 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.847122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:37.848500 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:37.848503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:37.848869 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:37.848871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:37.849025 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:37.849633 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:37.849646 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:37.851527 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:37.853289 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:37.853298 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:37.853300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:37.853643 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:37.853849 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:37.853850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:37.853851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:37.853991 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:37.854303 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:37.854310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:37.854445 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 07:28:37.854535 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 07:28:37.854669 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260810 07:28:37.855553 15190 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44883
I20260810 07:28:37.855670 15190 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:37.855870 15190 heartbeater.cc:511] Master 127.24.60.254:44883 requested a full tablet report, sending...
I20260810 07:28:37.856448 14809 ts_manager.cc:194] Registered new tserver with Master: 9dd9cf32fac14889be1aafcca8d69417 (127.24.60.194:41065)
I20260810 07:28:37.856923 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.194@KRBTEST.COM'} at 127.24.60.194:33619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
Entry for principal kudu/127.24.60.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab.
I20260810 07:28:37.875675 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44883
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:37.960287 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:37.960511 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:37.960587 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:37.961827 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:37.961874 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:37.961901 15195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:37.963316 15195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.195.keytab
--principal=kudu/127.24.60.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:44883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:37.963521 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:37.963711 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:37.969053 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:37.969087 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:37.969429 15195 server_base.cc:1034] running on GCE node
W20260810 07:28:37.969549 15202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:37.969780 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:37.970084 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:37.971278 15195 hybrid_clock.cc:648] HybridClock initialized: now 1786346917971260 us; error 52 us; skew 500 ppm
Aug 10 07:28:37 dist-test-slave-vxj2 krb5kdc[14756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:37.974051 15195 init.cc:377] Logged in from keytab as kudu/127.24.60.195@KRBTEST.COM (short username kudu)
I20260810 07:28:37.974568 15195 webserver.cc:460] Webserver started at http://127.24.60.195:38979/ using document root <none> and password file <none>
I20260810 07:28:37.974828 15195 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:37.974908 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:37.975060 15195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:37.976189 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "478c81c177b446d08e065052af322952"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "f87e1bb7334e3558132781007d656c8e"
server_key_iv: "1e3f9e18150c8178acbefba8f3d70843"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.976562 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "478c81c177b446d08e065052af322952"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "f87e1bb7334e3558132781007d656c8e"
server_key_iv: "1e3f9e18150c8178acbefba8f3d70843"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.978024 15195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:37.978794 15215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.978977 15195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 07:28:37.979087 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "478c81c177b446d08e065052af322952"
format_stamp: "Formatted at 2026-08-10 07:28:37 on dist-test-slave-vxj2"
server_key: "f87e1bb7334e3558132781007d656c8e"
server_key_iv: "1e3f9e18150c8178acbefba8f3d70843"
server_key_version: "encryptionkey@0"
I20260810 07:28:37.979161 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:37.992802 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:37.994860 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:37.995046 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:37.995306 15195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:37.995702 15195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:37.995762 15195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:37.995834 15195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:37.995863 15195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.002664 15195 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:38329
I20260810 07:28:38.002712 15356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:38329 every 8 connection(s)
I20260810 07:28:38.003096 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:28:38 dist-test-slave-vxj2 krb5kdc[14756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.195@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:38.011562 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 15195
I20260810 07:28:38.011708 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:38.011988 24819 external_mini_cluster.cc:1428] Setting key d254319d19641f72390dab2a574f46a4
I20260810 07:28:38.014290 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.003999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:55289 (local address 127.24.60.254:44883)
0810 07:28:38.004076 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.004078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.004300 (+ 222us) server_negotiation.cc:407] Connection header received
0810 07:28:38.004881 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.004882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.004895 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.005021 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.006124 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.006647 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.007707 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.007916 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.008013 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.010622 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.010623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.010624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:38.010633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.012509 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:38.013242 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:38.013244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:38.013244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:38.013320 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:38.013625 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:38.013625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:38.013626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:38.013750 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.013802 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.014144 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.014223 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:38.014484 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.003982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:44883 (local address 127.24.60.195:55289)
0810 07:28:38.004170 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.004209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.004762 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.005109 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.005113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.005796 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:38.006006 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.006011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.006754 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.006755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.007607 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.007609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.008028 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.008029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.008161 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.008760 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:38.008769 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:38.010495 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:38.012642 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:38.012655 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:38.012658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:38.013098 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:38.013407 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:38.013408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:38.013408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:38.013553 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:38.013811 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:38.013818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:38.013995 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.014087 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.014238 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260810 07:28:38.015038 15357 heartbeater.cc:347] Connected to a master server at 127.24.60.254:44883
I20260810 07:28:38.015173 15357 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:38.015419 15357 heartbeater.cc:511] Master 127.24.60.254:44883 requested a full tablet report, sending...
I20260810 07:28:38.015842 14809 ts_manager.cc:194] Registered new tserver with Master: 478c81c177b446d08e065052af322952 (127.24.60.195:38329)
I20260810 07:28:38.016307 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.60.195@KRBTEST.COM'} at 127.24.60.195:55289
I20260810 07:28:38.024667 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:38.033207 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.026410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.24.60.254:44883)
0810 07:28:38.026491 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.026494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.026630 (+ 136us) server_negotiation.cc:407] Connection header received
0810 07:28:38.026693 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.026694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.026703 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.026753 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.027477 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.027967 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.028751 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.028943 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.028999 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.030405 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.030406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.030407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:38.030414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.031780 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:38.032160 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:38.032161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:38.032162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:38.032221 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:38.032413 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:38.032414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:38.032414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:38.032649 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.032729 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.032911 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.033001 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:38.035115 14809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:38.036799 14809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:38.044309 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.039536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:33577 (local address 127.0.0.1:36470)
0810 07:28:38.039693 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.039726 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.039826 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.040227 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.040230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.040233 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.040419 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.040423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.041851 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.041852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.043226 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.043227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.044074 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.044075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.044138 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.044141 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.044194 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.044230 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":103,"thread_start_us":88,"threads_started":1}
I20260810 07:28:38.044327 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.039678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.24.60.193:33577)
0810 07:28:38.039965 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.039971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.039982 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:28:38.040024 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.040027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.040043 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.040125 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.040552 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.041723 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.043301 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.044006 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.044076 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.044127 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.044205 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.044246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
I20260810 07:28:38.044414 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.040179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.24.60.194:41065)
0810 07:28:38.040339 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.040344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.040493 (+ 149us) server_negotiation.cc:407] Connection header received
0810 07:28:38.040599 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.040602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.040619 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.040692 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.040908 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.041705 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.043118 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.043857 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.043932 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.043990 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.044277 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.044322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260810 07:28:38.044425 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.039913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.24.60.195:38329)
0810 07:28:38.040132 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.040136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.040149 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:28:38.040226 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.040228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.040243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.040319 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.040558 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.041723 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.043325 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.043821 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.043913 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.043975 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.044279 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.044335 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":137,"threads_started":1}
I20260810 07:28:38.044234 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.039785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:38329 (local address 127.0.0.1:53942)
0810 07:28:38.040087 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.040098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.040153 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.040401 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.040402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.040404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.040438 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.040476 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.041851 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.041852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.043210 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.043214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.043914 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.043915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.043995 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.044008 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.044076 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.044134 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":96,"thread_start_us":73,"threads_started":1}
I20260810 07:28:38.045289 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.039957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:41065 (local address 127.0.0.1:40506)
0810 07:28:38.040358 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.040368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.040425 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.040775 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.040776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.040778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.040812 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.040813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.041814 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.041816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.043019 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.043022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.044030 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.044031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.044102 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.044106 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.044156 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.044205 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":86,"thread_start_us":76,"threads_started":1}
I20260810 07:28:38.045724 14958 tablet_service.cc:1467] Processing CreateTablet for tablet a26d5ed2fa184017a9cf14b4053b7e36 (DEFAULT_TABLE table=test-table [id=8836e92582884411bf26de181bc2ddc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:38.045728 15275 tablet_service.cc:1467] Processing CreateTablet for tablet a26d5ed2fa184017a9cf14b4053b7e36 (DEFAULT_TABLE table=test-table [id=8836e92582884411bf26de181bc2ddc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:38.046121 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26d5ed2fa184017a9cf14b4053b7e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.046121 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26d5ed2fa184017a9cf14b4053b7e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.046114 15109 tablet_service.cc:1467] Processing CreateTablet for tablet a26d5ed2fa184017a9cf14b4053b7e36 (DEFAULT_TABLE table=test-table [id=8836e92582884411bf26de181bc2ddc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:38.046461 15109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26d5ed2fa184017a9cf14b4053b7e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.049109 15374 tablet_bootstrap.cc:492] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Bootstrap starting.
I20260810 07:28:38.049619 15380 tablet_bootstrap.cc:492] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: Bootstrap starting.
I20260810 07:28:38.049942 15374 tablet_bootstrap.cc:654] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.050312 15380 tablet_bootstrap.cc:654] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.050593 15374 log.cc:824] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.050899 15380 log.cc:824] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.051308 15379 tablet_bootstrap.cc:492] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Bootstrap starting.
I20260810 07:28:38.051503 15374 tablet_bootstrap.cc:492] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: No bootstrap required, opened a new log
I20260810 07:28:38.051584 15374 ts_tablet_manager.cc:1397] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 07:28:38.051748 15380 tablet_bootstrap.cc:492] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: No bootstrap required, opened a new log
I20260810 07:28:38.051831 15380 ts_tablet_manager.cc:1397] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:38.051970 15379 tablet_bootstrap.cc:654] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.052524 15379 log.cc:824] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.053370 15379 tablet_bootstrap.cc:492] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: No bootstrap required, opened a new log
I20260810 07:28:38.053437 15379 ts_tablet_manager.cc:1397] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:38.053864 15380 raft_consensus.cc:348] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.053870 15374 raft_consensus.cc:348] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.054044 15374 raft_consensus.cc:374] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.054044 15380 raft_consensus.cc:374] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.054085 15374 raft_consensus.cc:729] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd9cf32fac14889be1aafcca8d69417, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.054087 15380 raft_consensus.cc:729] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430b6f58602e41a0ac179e1ca5f5aa33, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.054241 15380 consensus_queue.cc:260] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.054241 15374 consensus_queue.cc:260] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.054550 15380 ts_tablet_manager.cc:1428] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:38.054551 15374 ts_tablet_manager.cc:1428] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 07:28:38.054697 15033 heartbeater.cc:503] Master 127.24.60.254:44883 was elected leader, sending a full tablet report...
I20260810 07:28:38.054975 15190 heartbeater.cc:503] Master 127.24.60.254:44883 was elected leader, sending a full tablet report...
I20260810 07:28:38.055555 15379 raft_consensus.cc:348] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.055760 15379 raft_consensus.cc:374] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.055809 15379 raft_consensus.cc:729] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478c81c177b446d08e065052af322952, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.055932 15379 consensus_queue.cc:260] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.056197 15379 ts_tablet_manager.cc:1428] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:38.056259 15357 heartbeater.cc:503] Master 127.24.60.254:44883 was elected leader, sending a full tablet report...
W20260810 07:28:38.093845 15191 tablet.cc:2369] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:38.171269 15034 tablet.cc:2369] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:38.196851 15384 raft_consensus.cc:482] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:38.196997 15384 raft_consensus.cc:504] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.197456 15384 leader_election.cc:288] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 478c81c177b446d08e065052af322952 (127.24.60.195:38329), 9dd9cf32fac14889be1aafcca8d69417 (127.24.60.194:41065)
I20260810 07:28:38.200743 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.197753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:37107 (local address 127.24.60.195:38329)
0810 07:28:38.197850 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.197852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.197970 (+ 118us) server_negotiation.cc:407] Connection header received
0810 07:28:38.198035 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.198036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.198050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.198121 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.198345 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.198823 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.199794 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.200395 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.200474 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.200483 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.200626 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.200673 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 07:28:38.200742 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.197670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:38329 (local address 127.24.60.193:37107)
0810 07:28:38.197879 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.197891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.197972 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.198157 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.198158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.198161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.198251 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.198253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.198911 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.198912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.199694 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.199696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.200483 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.200484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.200541 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.200545 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.200594 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.200627 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260810 07:28:38.201050 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.198071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:47119 (local address 127.24.60.194:41065)
0810 07:28:38.198156 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.198158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.198171 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:28:38.198212 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.198215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.198228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.198289 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.198522 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.199011 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.199989 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.200601 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.200673 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.200681 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.200931 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.200982 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:28:38.201083 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26d5ed2fa184017a9cf14b4053b7e36" candidate_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478c81c177b446d08e065052af322952" is_pre_election: true
I20260810 07:28:38.201345 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.197941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:41065 (local address 127.24.60.193:47119)
0810 07:28:38.198069 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.198084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.198150 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.198309 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.198310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.198312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.198402 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.198404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.199108 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.199109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.199873 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.199875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.200689 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.200690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.200759 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.200764 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.201184 (+ 420us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.201234 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260810 07:28:38.201350 15301 raft_consensus.cc:2454] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430b6f58602e41a0ac179e1ca5f5aa33 in term 0.
I20260810 07:28:38.201615 14904 leader_election.cc:302] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430b6f58602e41a0ac179e1ca5f5aa33, 478c81c177b446d08e065052af322952; no voters:
I20260810 07:28:38.201725 15133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26d5ed2fa184017a9cf14b4053b7e36" candidate_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd9cf32fac14889be1aafcca8d69417" is_pre_election: true
I20260810 07:28:38.201771 15384 raft_consensus.cc:2781] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:38.201831 15384 raft_consensus.cc:482] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:38.201862 15384 raft_consensus.cc:3037] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.201882 15133 raft_consensus.cc:2454] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430b6f58602e41a0ac179e1ca5f5aa33 in term 0.
I20260810 07:28:38.202647 15384 raft_consensus.cc:504] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.202762 15384 leader_election.cc:288] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [CANDIDATE]: Term 1 election: Requested vote from peers 478c81c177b446d08e065052af322952 (127.24.60.195:38329), 9dd9cf32fac14889be1aafcca8d69417 (127.24.60.194:41065)
I20260810 07:28:38.203001 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26d5ed2fa184017a9cf14b4053b7e36" candidate_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478c81c177b446d08e065052af322952"
I20260810 07:28:38.203058 15133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26d5ed2fa184017a9cf14b4053b7e36" candidate_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd9cf32fac14889be1aafcca8d69417"
I20260810 07:28:38.203099 15301 raft_consensus.cc:3037] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.203178 15133 raft_consensus.cc:3037] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.203907 15301 raft_consensus.cc:2454] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430b6f58602e41a0ac179e1ca5f5aa33 in term 1.
I20260810 07:28:38.204041 15133 raft_consensus.cc:2454] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430b6f58602e41a0ac179e1ca5f5aa33 in term 1.
I20260810 07:28:38.204089 14904 leader_election.cc:302] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430b6f58602e41a0ac179e1ca5f5aa33, 478c81c177b446d08e065052af322952; no voters:
I20260810 07:28:38.204200 15384 raft_consensus.cc:2781] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:38.204396 15384 raft_consensus.cc:686] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 1 LEADER]: Becoming Leader. State: Replica: 430b6f58602e41a0ac179e1ca5f5aa33, State: Running, Role: LEADER
I20260810 07:28:38.204509 15384 consensus_queue.cc:237] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } }
I20260810 07:28:38.205513 14808 catalog_manager.cc:5495] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 430b6f58602e41a0ac179e1ca5f5aa33 (127.24.60.193). New cstate: current_term: 1 leader_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "430b6f58602e41a0ac179e1ca5f5aa33" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 33577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 } health_report { overall_health: UNKNOWN } } }
I20260810 07:28:38.228685 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.226035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.24.60.193:33577)
0810 07:28:38.226122 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.226129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.226138 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:28:38.226181 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.226182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.226202 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.226273 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:38.226458 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.227097 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.227973 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.228162 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.228218 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.228360 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:38.228467 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:38.228529 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.228560 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.228579 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:28:38.230996 15133 raft_consensus.cc:1261] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 1 FOLLOWER]: Refusing update from remote peer 430b6f58602e41a0ac179e1ca5f5aa33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:38.231005 15301 raft_consensus.cc:1261] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 1 FOLLOWER]: Refusing update from remote peer 430b6f58602e41a0ac179e1ca5f5aa33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:38.231451 15389 consensus_queue.cc:1035] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "478c81c177b446d08e065052af322952" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 38329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:38.231601 15384 consensus_queue.cc:1035] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd9cf32fac14889be1aafcca8d69417" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 41065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:38.234333 15394 mvcc.cc:205] Tried to move back new op lower bound from 7316876977070350336 to 7316876976966516736. Current Snapshot: MvccSnapshot[applied={T|T < 7316876977070350336}]
I20260810 07:28:38.234856 15395 mvcc.cc:205] Tried to move back new op lower bound from 7316876977070350336 to 7316876976966516736. Current Snapshot: MvccSnapshot[applied={T|T < 7316876977070350336}]
I20260810 07:28:38.238020 14809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:38.238236 14809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:38.239151 14809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf46352f9b6e4556855ec65787ba90b3: Sending DeleteTablet for 3 replicas of tablet a26d5ed2fa184017a9cf14b4053b7e36
I20260810 07:28:38.239545 14958 tablet_service.cc:1514] Processing DeleteTablet for tablet a26d5ed2fa184017a9cf14b4053b7e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:38 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:36470
I20260810 07:28:38.239583 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet a26d5ed2fa184017a9cf14b4053b7e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:38 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:53942
I20260810 07:28:38.239893 15109 tablet_service.cc:1514] Processing DeleteTablet for tablet a26d5ed2fa184017a9cf14b4053b7e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:38 UTC) from {username='kudu', principal='kudu/127.24.60.254@KRBTEST.COM'} at 127.0.0.1:40506
I20260810 07:28:38.240156 15405 tablet_replica.cc:330] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33: stopping tablet replica
I20260810 07:28:38.240195 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 14884
I20260810 07:28:38.240271 15405 raft_consensus.cc:2229] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:28:38.240411 15405 raft_consensus.cc:2258] T a26d5ed2fa184017a9cf14b4053b7e36 P 430b6f58602e41a0ac179e1ca5f5aa33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:38.244047 15406 tablet_replica.cc:330] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: stopping tablet replica
I20260810 07:28:38.244204 15406 raft_consensus.cc:2229] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:38.244338 15406 raft_consensus.cc:2258] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:38.244486 15407 tablet_replica.cc:330] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: stopping tablet replica
I20260810 07:28:38.244567 15407 raft_consensus.cc:2229] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:38.244681 15407 raft_consensus.cc:2258] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:38.245081 15406 ts_tablet_manager.cc:1905] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:38.245096 15407 ts_tablet_manager.cc:1905] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:38.247048 15406 ts_tablet_manager.cc:1918] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:38.247118 15406 log.cc:1196] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/a26d5ed2fa184017a9cf14b4053b7e36
I20260810 07:28:38.247339 15407 ts_tablet_manager.cc:1918] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:38.247424 15407 log.cc:1196] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346663269703-24819-0/minicluster-data/ts-1/wal/wals/a26d5ed2fa184017a9cf14b4053b7e36
I20260810 07:28:38.247488 15406 ts_tablet_manager.cc:1939] T a26d5ed2fa184017a9cf14b4053b7e36 P 478c81c177b446d08e065052af322952: Deleting consensus metadata
I20260810 07:28:38.247713 15407 ts_tablet_manager.cc:1939] T a26d5ed2fa184017a9cf14b4053b7e36 P 9dd9cf32fac14889be1aafcca8d69417: Deleting consensus metadata
I20260810 07:28:38.248062 14793 catalog_manager.cc:4841] TS 478c81c177b446d08e065052af322952 (127.24.60.195:38329): tablet a26d5ed2fa184017a9cf14b4053b7e36 (table test-table [id=8836e92582884411bf26de181bc2ddc7]) successfully deleted
W20260810 07:28:38.248361 14793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:33577 (error 108)
I20260810 07:28:38.248657 14793 catalog_manager.cc:4841] TS 9dd9cf32fac14889be1aafcca8d69417 (127.24.60.194:41065): tablet a26d5ed2fa184017a9cf14b4053b7e36 (table test-table [id=8836e92582884411bf26de181bc2ddc7]) successfully deleted
I20260810 07:28:38.248658 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 15038
I20260810 07:28:38.250113 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.248820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:33577 (local address 127.0.0.1:36496)
0810 07:28:38.250029 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.250080 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:33577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1179}
W20260810 07:28:38.250267 14793 catalog_manager.cc:4584] TS 430b6f58602e41a0ac179e1ca5f5aa33 (127.24.60.193:33577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a26d5ed2fa184017a9cf14b4053b7e36: Network error: Client connection negotiation failed: client connection to 127.24.60.193:33577: connect: Connection refused (error 111)
I20260810 07:28:38.254743 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 15195
I20260810 07:28:38.263832 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 14772
2026-08-10T07:28:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2196 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T07:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:38Z Disabled control of system clock
I20260810 07:28:38.296023 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:37367
--webserver_interface=127.24.60.254
--webserver_port=0
--builtin_ntp_servers=127.24.60.212:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:38.381047 15413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:38.381371 15413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:38.381397 15413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:38.382666 15413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:38.382714 15413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:38.382727 15413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:38.382740 15413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:38.382752 15413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:38.384110 15413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:37367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:38.384325 15413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:38.384588 15413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:38.389978 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.389978 15424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.390005 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:38.390604 15413 server_base.cc:1034] running on GCE node
I20260810 07:28:38.390894 15413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:38.391261 15413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:38.392463 15413 hybrid_clock.cc:648] HybridClock initialized: now 1786346918392362 us; error 116 us; skew 500 ppm
I20260810 07:28:38.394265 15413 webserver.cc:460] Webserver started at http://127.24.60.254:39663/ using document root <none> and password file <none>
I20260810 07:28:38.394598 15413 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:38.394654 15413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:38.394841 15413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:38.396107 15413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "e9b9150dc9dd4f459ecc6302a0df453e"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "a6bdf58102c6435081d8cc49291da124"
server_key_iv: "4df6643f1dfae6049abcf9182271a493"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.396529 15413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "e9b9150dc9dd4f459ecc6302a0df453e"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "a6bdf58102c6435081d8cc49291da124"
server_key_iv: "4df6643f1dfae6049abcf9182271a493"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.398309 15413 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 07:28:38.399241 15431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.399485 15413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:38.399611 15413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "e9b9150dc9dd4f459ecc6302a0df453e"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "a6bdf58102c6435081d8cc49291da124"
server_key_iv: "4df6643f1dfae6049abcf9182271a493"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.399701 15413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:38.420845 15413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:38.421324 15413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:38.421517 15413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:38.426091 15413 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:37367
I20260810 07:28:38.426162 15494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:37367 every 8 connection(s)
I20260810 07:28:38.426607 15413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:38.427664 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.430052 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e: Bootstrap starting.
I20260810 07:28:38.430612 15496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.430876 15496 log.cc:824] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.431542 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 15413
I20260810 07:28:38.431658 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:38.431727 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e: No bootstrap required, opened a new log
I20260810 07:28:38.431869 24819 external_mini_cluster.cc:1428] Setting key 8c97dfab28ec697aabf2e66303378b0e
I20260810 07:28:38.434616 15496 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37367 } }
I20260810 07:28:38.434818 15496 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.434860 15496 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b9150dc9dd4f459ecc6302a0df453e, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.434994 15496 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37367 } }
I20260810 07:28:38.435077 15496 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:38.435119 15496 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:38.435165 15496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.435913 15496 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37367 } }
I20260810 07:28:38.436013 15496 leader_election.cc:302] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b9150dc9dd4f459ecc6302a0df453e; no voters:
I20260810 07:28:38.436268 15496 leader_election.cc:288] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:38.436299 15501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:38.436441 15501 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [term 1 LEADER]: Becoming Leader. State: Replica: e9b9150dc9dd4f459ecc6302a0df453e, State: Running, Role: LEADER
I20260810 07:28:38.436563 15501 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37367 } }
I20260810 07:28:38.436573 15496 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:38.437361 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37367 } } }
I20260810 07:28:38.437451 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:38.437601 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b9150dc9dd4f459ecc6302a0df453e. Latest consensus state: current_term: 1 leader_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b9150dc9dd4f459ecc6302a0df453e" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37367 } } }
I20260810 07:28:38.437668 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:38.437800 15510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:38.438299 15510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:38.438616 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.432463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.24.60.254:37367)
0810 07:28:38.432698 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.432706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.432735 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:28:38.433241 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.433249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.433315 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.433440 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.433705 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.434633 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.435738 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.436241 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.436379 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.437277 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.437282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.437283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:38.437659 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.437784 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:38.437802 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.437975 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.438200 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.438298 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":119,"threads_started":1}
I20260810 07:28:38.439942 15510 catalog_manager.cc:1347] Generated new cluster ID: d1e258ba243548289d5b198d7134b85f
I20260810 07:28:38.439999 15510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:38.456079 15510 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:38.456745 15510 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:38.463138 15510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e: Generated new TSK 0
I20260810 07:28:38.463385 15510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:38.467636 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.193:0
--local_ip_for_outbound_sockets=127.24.60.193
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37367
--builtin_ntp_servers=127.24.60.212:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:38.553750 15521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:38.554009 15521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:38.554052 15521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:38.555271 15521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:38.555341 15521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:38.555385 15521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.193
I20260810 07:28:38.556810 15521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.60.193
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.193
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:38.557101 15521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:38.557457 15521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:38.563683 15529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.563705 15530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:38.563990 15521 server_base.cc:1034] running on GCE node
W20260810 07:28:38.563709 15532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:38.564278 15521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:38.564563 15521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:38.565732 15521 hybrid_clock.cc:648] HybridClock initialized: now 1786346918565719 us; error 31 us; skew 500 ppm
I20260810 07:28:38.567096 15521 webserver.cc:460] Webserver started at http://127.24.60.193:43147/ using document root <none> and password file <none>
I20260810 07:28:38.567365 15521 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:38.567440 15521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:38.567627 15521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:38.568769 15521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data/instance:
uuid: "9018837161ff4891bb62c38d95d2d38f"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "2a63d2827e9d7d161318d9327c56e560"
server_key_iv: "ecf221e5a9d6cfca46d8520115b9fa6f"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.569205 15521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance:
uuid: "9018837161ff4891bb62c38d95d2d38f"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "2a63d2827e9d7d161318d9327c56e560"
server_key_iv: "ecf221e5a9d6cfca46d8520115b9fa6f"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.570690 15521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:38.571491 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.571703 15521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:38.571808 15521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal
uuid: "9018837161ff4891bb62c38d95d2d38f"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "2a63d2827e9d7d161318d9327c56e560"
server_key_iv: "ecf221e5a9d6cfca46d8520115b9fa6f"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.571893 15521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:38.580581 15521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:38.580924 15521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:38.581087 15521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:38.581375 15521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:38.581745 15521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:38.581802 15521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.581867 15521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:38.581915 15521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.588009 15521 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.193:35289
I20260810 07:28:38.588095 15669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.193:35289 every 8 connection(s)
I20260810 07:28:38.588461 15521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:38.593439 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 15521
I20260810 07:28:38.593585 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:38.593799 24819 external_mini_cluster.cc:1428] Setting key 0049f8a854b7573c3932f318567ccf4a
I20260810 07:28:38.594408 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.589365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:59119 (local address 127.24.60.254:37367)
0810 07:28:38.589433 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.589435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.589735 (+ 300us) server_negotiation.cc:407] Connection header received
0810 07:28:38.590322 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.590323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.590344 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.590399 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.591117 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.591644 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.592656 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.592862 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.592961 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.594008 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.594009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.594010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:38.594014 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.594047 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:38.594053 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.594116 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.594293 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.594322 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:28:38.594647 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.589374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:37367 (local address 127.24.60.193:59119)
0810 07:28:38.589609 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.589645 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.590199 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.590486 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.590490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.590498 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:38.591014 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.591019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.591755 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.591756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.592560 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.592562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.592970 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.592972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.593108 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.593787 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:38.593788 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:38.593819 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:38.593843 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:38.594170 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:38.594179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:38.594183 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.594280 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.594390 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260810 07:28:38.595065 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.194:0
--local_ip_for_outbound_sockets=127.24.60.194
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37367
--builtin_ntp_servers=127.24.60.212:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:28:38.595170 15671 heartbeater.cc:347] Connected to a master server at 127.24.60.254:37367
I20260810 07:28:38.595281 15671 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:38.595495 15671 heartbeater.cc:511] Master 127.24.60.254:37367 requested a full tablet report, sending...
I20260810 07:28:38.596379 15448 ts_manager.cc:194] Registered new tserver with Master: 9018837161ff4891bb62c38d95d2d38f (127.24.60.193:35289)
I20260810 07:28:38.597394 15448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.193:59119
W20260810 07:28:38.689785 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:38.690023 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:38.690048 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:38.691152 15674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:38.691186 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:38.691216 15674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.194
I20260810 07:28:38.692525 15674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.60.194
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.194
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:38.692716 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:38.692973 15674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:38.698096 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.698097 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.698134 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:38.698462 15674 server_base.cc:1034] running on GCE node
I20260810 07:28:38.698698 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:38.698992 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:38.700158 15674 hybrid_clock.cc:648] HybridClock initialized: now 1786346918700120 us; error 54 us; skew 500 ppm
I20260810 07:28:38.701624 15674 webserver.cc:460] Webserver started at http://127.24.60.194:43031/ using document root <none> and password file <none>
I20260810 07:28:38.701881 15674 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:38.702003 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:38.702165 15674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:38.703332 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data/instance:
uuid: "c094275fe2864af7bb564fee92417f7c"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "a5b9e2d80a6bfa51e5590bbbb8474db5"
server_key_iv: "dce9fa6ee1b26c6a5ac3fa9d8ce921b9"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.703708 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance:
uuid: "c094275fe2864af7bb564fee92417f7c"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "a5b9e2d80a6bfa51e5590bbbb8474db5"
server_key_iv: "dce9fa6ee1b26c6a5ac3fa9d8ce921b9"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.705086 15674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 07:28:38.705938 15689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.706131 15674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:38.706241 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal
uuid: "c094275fe2864af7bb564fee92417f7c"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "a5b9e2d80a6bfa51e5590bbbb8474db5"
server_key_iv: "dce9fa6ee1b26c6a5ac3fa9d8ce921b9"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.706316 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:38.716337 15674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:38.716679 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:38.716837 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:38.717082 15674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:38.717499 15674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:38.717566 15674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.717609 15674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:38.717641 15674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.723817 15674 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.194:44921
I20260810 07:28:38.723969 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.194:44921 every 8 connection(s)
I20260810 07:28:38.724272 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:38.730693 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.725217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.194:57291 (local address 127.24.60.254:37367)
0810 07:28:38.725281 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.725284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.725573 (+ 289us) server_negotiation.cc:407] Connection header received
0810 07:28:38.726284 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.726286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.726303 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.726411 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.727492 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.728006 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.729111 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.729429 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.729492 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.730418 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.730419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.730436 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:38.730442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.730457 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:38.730461 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.730512 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.730606 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.730629 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 07:28:38.731056 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.725156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:37367 (local address 127.24.60.194:57291)
0810 07:28:38.725437 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.725474 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.726114 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.726596 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.726601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.726613 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:38.727388 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.727394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.728121 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.728121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.729017 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.729019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.729511 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.729512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.729628 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.730257 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:38.730258 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:38.730292 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:38.730318 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:38.730528 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:38.730533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:38.730535 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.730604 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.730701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260810 07:28:38.731643 15818 heartbeater.cc:347] Connected to a master server at 127.24.60.254:37367
I20260810 07:28:38.731748 15818 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:38.731906 15818 heartbeater.cc:511] Master 127.24.60.254:37367 requested a full tablet report, sending...
I20260810 07:28:38.732347 15448 ts_manager.cc:194] Registered new tserver with Master: c094275fe2864af7bb564fee92417f7c (127.24.60.194:44921)
I20260810 07:28:38.732694 15448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.194:57291
I20260810 07:28:38.732905 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 15674
I20260810 07:28:38.733067 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:38.733340 24819 external_mini_cluster.cc:1428] Setting key 8f93c8f22041d07bcf732191926d679f
I20260810 07:28:38.734385 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.60.195:0
--local_ip_for_outbound_sockets=127.24.60.195
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37367
--builtin_ntp_servers=127.24.60.212:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:28:38.819530 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:38.819749 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:38.819779 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:38.821010 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:38.821056 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:38.821081 15822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.60.195
I20260810 07:28:38.822556 15822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.60.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.60.195
--webserver_port=0
--tserver_master_addrs=127.24.60.254:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.60.195
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:38.822827 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:38.823145 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:38.828249 15828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.828276 15830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:38.828279 15832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:38.828521 15822 server_base.cc:1034] running on GCE node
I20260810 07:28:38.828742 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:38.829043 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:38.830214 15822 hybrid_clock.cc:648] HybridClock initialized: now 1786346918830209 us; error 61 us; skew 500 ppm
I20260810 07:28:38.831461 15822 webserver.cc:460] Webserver started at http://127.24.60.195:46313/ using document root <none> and password file <none>
I20260810 07:28:38.831718 15822 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:38.831787 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:38.831938 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:38.833060 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data/instance:
uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "b66aa95a8fa015ad8d2dd37d90c91754"
server_key_iv: "6c0ad8684c4c656fafcec0c79f99d8de"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.833503 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance:
uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "b66aa95a8fa015ad8d2dd37d90c91754"
server_key_iv: "6c0ad8684c4c656fafcec0c79f99d8de"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.834829 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:38.835673 15839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.835857 15822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:38.835979 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal
uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a"
format_stamp: "Formatted at 2026-08-10 07:28:38 on dist-test-slave-vxj2"
server_key: "b66aa95a8fa015ad8d2dd37d90c91754"
server_key_iv: "6c0ad8684c4c656fafcec0c79f99d8de"
server_key_version: "encryptionkey@0"
I20260810 07:28:38.836104 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:38.846858 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:38.847206 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:38.847368 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:38.847611 15822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:38.847980 15822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:38.848038 15822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.848083 15822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:38.848111 15822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:38.854266 15822 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.195:45009
I20260810 07:28:38.854322 15988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.195:45009 every 8 connection(s)
I20260810 07:28:38.854686 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:38.860282 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 15822
I20260810 07:28:38.860428 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:38.860585 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.855584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.195:44579 (local address 127.24.60.254:37367)
0810 07:28:38.855659 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.855662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.855887 (+ 225us) server_negotiation.cc:407] Connection header received
0810 07:28:38.856501 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.856502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.856517 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.856564 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.857361 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.857877 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.858942 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.859157 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.859240 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.860244 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.860245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.860246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:38.860251 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.860268 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:38.860273 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.860338 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.860477 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.860521 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:28:38.860666 24819 external_mini_cluster.cc:1428] Setting key 9c408370a58a3f87a707f957bae33d7e
I20260810 07:28:38.860846 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.855465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:37367 (local address 127.24.60.195:44579)
0810 07:28:38.855736 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.855785 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.856361 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.856635 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.856640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.856650 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:38.857256 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.857263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.857994 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.857996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.858833 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.858835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.859249 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.859250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.859405 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.860024 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:28:38.860025 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:38.860086 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:28:38.860117 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:38.860360 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:38.860369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:38.860373 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.860476 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.860603 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
I20260810 07:28:38.861392 15990 heartbeater.cc:347] Connected to a master server at 127.24.60.254:37367
I20260810 07:28:38.861495 15990 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:38.861742 15990 heartbeater.cc:511] Master 127.24.60.254:37367 requested a full tablet report, sending...
I20260810 07:28:38.862169 15448 ts_manager.cc:194] Registered new tserver with Master: 14ffc4bfe9fb41c6a3fa1d4faa54b50a (127.24.60.195:45009)
I20260810 07:28:38.862560 15448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.60.195:44579
I20260810 07:28:38.872677 24819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:38.877403 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.874328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.24.60.254:37367)
0810 07:28:38.874409 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.874411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.874514 (+ 103us) server_negotiation.cc:407] Connection header received
0810 07:28:38.874560 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.874561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.874571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.874648 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:38.874826 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.875439 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.876255 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.876441 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.876498 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.877108 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:38.877109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:38.877110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:38.877114 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:38.877128 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:38.877132 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:38.877219 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.877270 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.877291 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:28:38.878961 15448 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:38.880599 15448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:38.887630 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.883005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:35289 (local address 127.0.0.1:35734)
0810 07:28:38.883288 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.883300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.883356 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.883634 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.883636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.883638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.883694 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.883696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.884955 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.884955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.886251 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.886254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.887305 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.887306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.887385 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.887401 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.887485 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.887546 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
I20260810 07:28:38.887667 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.883248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.24.60.193:35289)
0810 07:28:38.883471 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.883475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.883485 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:38.883526 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.883528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.883544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.883633 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.883765 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.884869 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.886357 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.887195 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.887279 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.887390 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.887520 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.887572 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260810 07:28:38.887940 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.883059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.24.60.195:45009)
0810 07:28:38.883214 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.883219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.883231 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:28:38.883372 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.883375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.883394 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.883514 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.883696 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.884874 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.886542 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.887357 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.887430 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.887553 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.887784 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.887835 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260810 07:28:38.888236 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.882972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:45009 (local address 127.0.0.1:33138)
0810 07:28:38.883112 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.883156 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.883223 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.883525 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.883528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.883530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.883607 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.883609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.885233 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.885234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.886370 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.886373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.887594 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.887596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.887668 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.887672 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.888077 (+ 405us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.888135 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260810 07:28:38.888674 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.883222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44921 (local address 127.0.0.1:50156)
0810 07:28:38.883838 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.883848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.883894 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.885245 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.885246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.885248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.885287 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.885289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.886573 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.886573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.887359 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.887361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.888432 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.888433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.888493 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.888498 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.888558 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.888605 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":576,"threads_started":1}
I20260810 07:28:38.888890 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.883570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.24.60.194:44921)
0810 07:28:38.885049 (+ 1479us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.885053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.885061 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:28:38.885094 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.885095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.885107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.885243 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.885358 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.886468 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.887456 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.888209 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.888286 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.888381 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.888588 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.888635 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":110,"threads_started":1}
I20260810 07:28:38.889074 15593 tablet_service.cc:1467] Processing CreateTablet for tablet 80d022351cfe40c08540c465773d6ced (DEFAULT_TABLE table=test-table [id=220b837131e444899b1e3a499acc845e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:38.889140 15910 tablet_service.cc:1467] Processing CreateTablet for tablet 80d022351cfe40c08540c465773d6ced (DEFAULT_TABLE table=test-table [id=220b837131e444899b1e3a499acc845e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:38.889508 15593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d022351cfe40c08540c465773d6ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.889559 15910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d022351cfe40c08540c465773d6ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.889585 15742 tablet_service.cc:1467] Processing CreateTablet for tablet 80d022351cfe40c08540c465773d6ced (DEFAULT_TABLE table=test-table [id=220b837131e444899b1e3a499acc845e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:38.889900 15742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d022351cfe40c08540c465773d6ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:38.892551 16006 tablet_bootstrap.cc:492] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f: Bootstrap starting.
I20260810 07:28:38.893313 16006 tablet_bootstrap.cc:654] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.893286 16009 tablet_bootstrap.cc:492] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: Bootstrap starting.
I20260810 07:28:38.893697 16006 log.cc:824] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.894033 16009 tablet_bootstrap.cc:654] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.894389 16009 log.cc:824] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.894708 16006 tablet_bootstrap.cc:492] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f: No bootstrap required, opened a new log
I20260810 07:28:38.894793 16006 ts_tablet_manager.cc:1397] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 07:28:38.894788 16008 tablet_bootstrap.cc:492] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Bootstrap starting.
I20260810 07:28:38.895364 16009 tablet_bootstrap.cc:492] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: No bootstrap required, opened a new log
I20260810 07:28:38.895460 16009 ts_tablet_manager.cc:1397] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:28:38.895568 16008 tablet_bootstrap.cc:654] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:38.896109 16008 log.cc:824] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:38.896932 16008 tablet_bootstrap.cc:492] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: No bootstrap required, opened a new log
I20260810 07:28:38.897014 16008 ts_tablet_manager.cc:1397] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:38.897742 16009 raft_consensus.cc:348] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.897912 16009 raft_consensus.cc:374] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.897958 16009 raft_consensus.cc:729] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c094275fe2864af7bb564fee92417f7c, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.898068 16009 consensus_queue.cc:260] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.898010 16006 raft_consensus.cc:348] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.898200 16006 raft_consensus.cc:374] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.898236 16006 raft_consensus.cc:729] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9018837161ff4891bb62c38d95d2d38f, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.898329 16009 ts_tablet_manager.cc:1428] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 07:28:38.898381 16006 consensus_queue.cc:260] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.898487 15818 heartbeater.cc:503] Master 127.24.60.254:37367 was elected leader, sending a full tablet report...
I20260810 07:28:38.898672 16006 ts_tablet_manager.cc:1428] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260810 07:28:38.898727 15671 heartbeater.cc:503] Master 127.24.60.254:37367 was elected leader, sending a full tablet report...
I20260810 07:28:38.900331 16008 raft_consensus.cc:348] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.900481 16008 raft_consensus.cc:374] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:38.900506 16008 raft_consensus.cc:729] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ffc4bfe9fb41c6a3fa1d4faa54b50a, State: Initialized, Role: FOLLOWER
I20260810 07:28:38.900628 16008 consensus_queue.cc:260] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.900892 16008 ts_tablet_manager.cc:1428] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260810 07:28:38.900960 15990 heartbeater.cc:503] Master 127.24.60.254:37367 was elected leader, sending a full tablet report...
I20260810 07:28:38.926688 16014 raft_consensus.cc:482] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:38.926822 16014 raft_consensus.cc:504] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.927235 16014 leader_election.cc:288] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ffc4bfe9fb41c6a3fa1d4faa54b50a (127.24.60.195:45009), c094275fe2864af7bb564fee92417f7c (127.24.60.194:44921)
I20260810 07:28:38.930279 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.927414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.195:45009 (local address 127.24.60.193:45385)
0810 07:28:38.927496 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.927509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.927557 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.927761 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.927762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.927765 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.927844 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.927846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.928519 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.928520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.929467 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.929470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.930014 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.930015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.930088 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.930092 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.930162 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.930205 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 07:28:38.930279 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.927524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:45385 (local address 127.24.60.195:45009)
0810 07:28:38.927608 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.927611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.927621 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:28:38.927659 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.927660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.927671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.927740 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.927923 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.928436 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.929562 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.929918 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.930013 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.930022 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.930169 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.930208 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:28:38.930579 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.927666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.60.193:33709 (local address 127.24.60.194:44921)
0810 07:28:38.927711 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.927713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.927840 (+ 127us) server_negotiation.cc:407] Connection header received
0810 07:28:38.927897 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.927898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.927910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.928015 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:38.928179 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.928767 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.929795 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.930153 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.930273 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.930282 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.930450 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.930509 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:28:38.930621 15931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d022351cfe40c08540c465773d6ced" candidate_uuid: "9018837161ff4891bb62c38d95d2d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" is_pre_election: true
I20260810 07:28:38.930823 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.927550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44921 (local address 127.24.60.193:33709)
0810 07:28:38.927761 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.927769 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:38.927842 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:38.928019 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:38.928019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:38.928021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:38.928105 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.928106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.928894 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.928895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.929697 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:38.929700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.930240 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.930241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:38.930348 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.930352 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:28:38.930681 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 07:28:38.930733 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":144,"threads_started":1}
I20260810 07:28:38.930811 15931 raft_consensus.cc:2454] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9018837161ff4891bb62c38d95d2d38f in term 0.
I20260810 07:28:38.931100 15541 leader_election.cc:302] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ffc4bfe9fb41c6a3fa1d4faa54b50a, 9018837161ff4891bb62c38d95d2d38f; no voters:
I20260810 07:28:38.931197 15767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d022351cfe40c08540c465773d6ced" candidate_uuid: "9018837161ff4891bb62c38d95d2d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094275fe2864af7bb564fee92417f7c" is_pre_election: true
I20260810 07:28:38.931241 16014 raft_consensus.cc:2781] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:38.931296 16014 raft_consensus.cc:482] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:38.931313 16014 raft_consensus.cc:3037] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.931380 15767 raft_consensus.cc:2454] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9018837161ff4891bb62c38d95d2d38f in term 0.
I20260810 07:28:38.932142 16014 raft_consensus.cc:504] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.932287 16014 leader_election.cc:288] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [CANDIDATE]: Term 1 election: Requested vote from peers 14ffc4bfe9fb41c6a3fa1d4faa54b50a (127.24.60.195:45009), c094275fe2864af7bb564fee92417f7c (127.24.60.194:44921)
I20260810 07:28:38.932493 15767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d022351cfe40c08540c465773d6ced" candidate_uuid: "9018837161ff4891bb62c38d95d2d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094275fe2864af7bb564fee92417f7c"
I20260810 07:28:38.932478 15931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d022351cfe40c08540c465773d6ced" candidate_uuid: "9018837161ff4891bb62c38d95d2d38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a"
I20260810 07:28:38.932565 15767 raft_consensus.cc:3037] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.932585 15931 raft_consensus.cc:3037] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:38.933385 15931 raft_consensus.cc:2454] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9018837161ff4891bb62c38d95d2d38f in term 1.
I20260810 07:28:38.933414 15767 raft_consensus.cc:2454] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9018837161ff4891bb62c38d95d2d38f in term 1.
I20260810 07:28:38.933602 15541 leader_election.cc:302] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9018837161ff4891bb62c38d95d2d38f, c094275fe2864af7bb564fee92417f7c; no voters:
I20260810 07:28:38.933725 16014 raft_consensus.cc:2781] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:38.933908 16014 raft_consensus.cc:686] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [term 1 LEADER]: Becoming Leader. State: Replica: 9018837161ff4891bb62c38d95d2d38f, State: Running, Role: LEADER
I20260810 07:28:38.934159 16014 consensus_queue.cc:237] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } }
I20260810 07:28:38.935066 15447 catalog_manager.cc:5495] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9018837161ff4891bb62c38d95d2d38f (127.24.60.193). New cstate: current_term: 1 leader_uuid: "9018837161ff4891bb62c38d95d2d38f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9018837161ff4891bb62c38d95d2d38f" member_type: VOTER last_known_addr { host: "127.24.60.193" port: 35289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 } health_report { overall_health: UNKNOWN } } }
I20260810 07:28:38.945751 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.943052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.24.60.193:35289)
0810 07:28:38.943130 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:38.943132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:38.943147 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:28:38.943229 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:38.943230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:38.943250 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:38.943328 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:38.943442 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.943908 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.944669 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:38.944867 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:38.944946 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:38.945137 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:38.945503 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:38.945579 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:38.945649 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 07:28:38.945670 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:28:38.947760 15931 raft_consensus.cc:1261] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 1 FOLLOWER]: Refusing update from remote peer 9018837161ff4891bb62c38d95d2d38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:38.947760 15767 raft_consensus.cc:1261] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 1 FOLLOWER]: Refusing update from remote peer 9018837161ff4891bb62c38d95d2d38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:38.948137 16014 consensus_queue.cc:1035] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ffc4bfe9fb41c6a3fa1d4faa54b50a" member_type: VOTER last_known_addr { host: "127.24.60.195" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:38.948326 16017 consensus_queue.cc:1035] T 80d022351cfe40c08540c465773d6ced P 9018837161ff4891bb62c38d95d2d38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c094275fe2864af7bb564fee92417f7c" member_type: VOTER last_known_addr { host: "127.24.60.194" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:38.951203 16021 mvcc.cc:205] Tried to move back new op lower bound from 7316876980007043072 to 7316876979955064832. Current Snapshot: MvccSnapshot[applied={T|T < 7316876980007043072}]
I20260810 07:28:38.954617 15447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:38.954730 15447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:38.955636 15447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b9150dc9dd4f459ecc6302a0df453e: Sending DeleteTablet for 3 replicas of tablet 80d022351cfe40c08540c465773d6ced
I20260810 07:28:38.956067 15910 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d022351cfe40c08540c465773d6ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:38 UTC) from {username='slave'} at 127.0.0.1:33138
I20260810 07:28:38.956243 15742 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d022351cfe40c08540c465773d6ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:38 UTC) from {username='slave'} at 127.0.0.1:50156
I20260810 07:28:38.956331 16032 tablet_replica.cc:330] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: stopping tablet replica
I20260810 07:28:38.956449 16032 raft_consensus.cc:2229] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:38.956581 16032 raft_consensus.cc:2258] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:38.956574 15593 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d022351cfe40c08540c465773d6ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:38 UTC) from {username='slave'} at 127.0.0.1:35734
I20260810 07:28:38.956597 16033 tablet_replica.cc:330] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: stopping tablet replica
I20260810 07:28:38.956812 16033 raft_consensus.cc:2229] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:38.956965 16033 raft_consensus.cc:2258] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:38.957098 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 15521
I20260810 07:28:38.957134 16032 ts_tablet_manager.cc:1905] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:38.960479 16032 ts_tablet_manager.cc:1918] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:38.960579 16032 log.cc:1196] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Deleting WAL directory at /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346663269703-24819-0/minicluster-data/ts-2/wal/wals/80d022351cfe40c08540c465773d6ced
I20260810 07:28:38.960879 16032 ts_tablet_manager.cc:1939] T 80d022351cfe40c08540c465773d6ced P 14ffc4bfe9fb41c6a3fa1d4faa54b50a: Deleting consensus metadata
I20260810 07:28:38.961397 15433 catalog_manager.cc:4841] TS 14ffc4bfe9fb41c6a3fa1d4faa54b50a (127.24.60.195:45009): tablet 80d022351cfe40c08540c465773d6ced (table test-table [id=220b837131e444899b1e3a499acc845e]) successfully deleted
W20260810 07:28:38.965526 15432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.60.193:35289 (error 108)
I20260810 07:28:38.965636 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 15674
I20260810 07:28:38.965821 16033 ts_tablet_manager.cc:1905] T 80d022351cfe40c08540c465773d6ced P c094275fe2864af7bb564fee92417f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:38.968641 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.966937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.193:35289 (local address 127.0.0.1:35748)
0810 07:28:38.968552 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.968604 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.193:35289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1588}
W20260810 07:28:38.968801 15432 catalog_manager.cc:4584] TS 9018837161ff4891bb62c38d95d2d38f (127.24.60.193:35289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80d022351cfe40c08540c465773d6ced: Network error: Client connection negotiation failed: client connection to 127.24.60.193:35289: connect: Connection refused (error 111)
I20260810 07:28:38.973608 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 15822
I20260810 07:28:38.973644 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:38.973513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.194:44921 (local address 127.0.0.1:50166)
0810 07:28:38.973577 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:38.973610 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.194:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 07:28:38.982801 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 15413
2026-08-10T07:28:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (718 ms)
[----------] 6 tests from EncryptionPolicyTest (9330 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:39 dist-test-slave-vxj2 krb5kdc[16039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:39 dist-test-slave-vxj2 krb5kdc[16039](info): set up 2 sockets
Aug 10 07:28:39 dist-test-slave-vxj2 krb5kdc[16039](info): commencing operation
krb5kdc: starting...
W20260810 07:28:40.112409 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:40 dist-test-slave-vxj2 krb5kdc[16039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:40.245519 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:40.334719 16056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:40.334961 16056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:40.334986 16056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:40.336251 16056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:40.336295 16056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:40.336309 16056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:40.336323 16056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:40.336335 16056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:40.337978 16056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:40.338266 16056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:40.338517 16056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:40.344015 16062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:40.344115 16056 server_base.cc:1034] running on GCE node
W20260810 07:28:40.344028 16063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:40.344040 16066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:40.344580 16056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:40.344862 16056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:40.346024 16056 hybrid_clock.cc:648] HybridClock initialized: now 1786346920345976 us; error 65 us; skew 500 ppm
Aug 10 07:28:40 dist-test-slave-vxj2 krb5kdc[16039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:40.352572 16056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:40.353052 16056 webserver.cc:460] Webserver started at http://127.0.0.1:38205/ using document root <none> and password file <none>
I20260810 07:28:40.353333 16056 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:40.353391 16056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:40.353506 16056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:40.354686 16056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "c1ce15938c4f4f218b912a950256faf9"
format_stamp: "Formatted at 2026-08-10 07:28:40 on dist-test-slave-vxj2"
server_key: "5109347c94b85c09f4fec98e606f1e1c"
server_key_iv: "6529e74279eb28ceeb50b2ab03f27fab"
server_key_version: "encryptionkey@0"
I20260810 07:28:40.355042 16056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "c1ce15938c4f4f218b912a950256faf9"
format_stamp: "Formatted at 2026-08-10 07:28:40 on dist-test-slave-vxj2"
server_key: "5109347c94b85c09f4fec98e606f1e1c"
server_key_iv: "6529e74279eb28ceeb50b2ab03f27fab"
server_key_version: "encryptionkey@0"
I20260810 07:28:40.356535 16056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:40.357448 16073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:40.357637 16056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:40.357762 16056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "c1ce15938c4f4f218b912a950256faf9"
format_stamp: "Formatted at 2026-08-10 07:28:40 on dist-test-slave-vxj2"
server_key: "5109347c94b85c09f4fec98e606f1e1c"
server_key_iv: "6529e74279eb28ceeb50b2ab03f27fab"
server_key_version: "encryptionkey@0"
I20260810 07:28:40.357849 16056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:40.373205 16056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:40.375743 16056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:40.375929 16056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:40.380424 16056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33471
I20260810 07:28:40.380525 16135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33471 every 8 connection(s)
I20260810 07:28:40.381104 16056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:40.382004 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16056
I20260810 07:28:40.382145 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:40.382340 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:40.382370 24819 external_mini_cluster.cc:1428] Setting key 7b231e56be927623ded4e3a44a453436
I20260810 07:28:40.385886 16136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9: Bootstrap starting.
I20260810 07:28:40.386585 16136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:40.386909 16136 log.cc:824] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:40.387801 16136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9: No bootstrap required, opened a new log
Aug 10 07:28:40 dist-test-slave-vxj2 krb5kdc[16039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:28:40.390857 16136 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ce15938c4f4f218b912a950256faf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33471 } }
I20260810 07:28:40.391053 16136 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:40.391101 16136 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ce15938c4f4f218b912a950256faf9, State: Initialized, Role: FOLLOWER
I20260810 07:28:40.391281 16136 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ce15938c4f4f218b912a950256faf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33471 } }
I20260810 07:28:40.391378 16136 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:40.391435 16136 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:40.391507 16136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:40.392160 16136 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ce15938c4f4f218b912a950256faf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33471 } }
I20260810 07:28:40.392258 16136 leader_election.cc:302] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1ce15938c4f4f218b912a950256faf9; no voters:
I20260810 07:28:40.392506 16136 leader_election.cc:288] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:40.392542 16142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:40.392654 16142 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [term 1 LEADER]: Becoming Leader. State: Replica: c1ce15938c4f4f218b912a950256faf9, State: Running, Role: LEADER
I20260810 07:28:40.392733 16142 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ce15938c4f4f218b912a950256faf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33471 } }
I20260810 07:28:40.392824 16136 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:40.393062 16142 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ce15938c4f4f218b912a950256faf9. Latest consensus state: current_term: 1 leader_uuid: "c1ce15938c4f4f218b912a950256faf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ce15938c4f4f218b912a950256faf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33471 } } }
I20260810 07:28:40.393137 16142 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:40.393409 16148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:40.394011 16148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:40.394403 16144 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1ce15938c4f4f218b912a950256faf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ce15938c4f4f218b912a950256faf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33471 } } }
I20260810 07:28:40.394834 16144 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:40.396245 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:40.382972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46499 (local address 127.0.0.1:33471)
0810 07:28:40.383192 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:40.383201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:40.383230 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:28:40.383929 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:40.383939 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:40.384006 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:40.384138 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:40.385017 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:40.386201 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:40.387315 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:40.388027 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:40.388123 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:40.391399 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:40.391404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:40.391405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:40.391421 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:40.394005 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:40.394702 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:40.394706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:40.394707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:40.394838 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:40.395146 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:40.395147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:40.395148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:40.395511 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:40.395703 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:40.395889 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 07:28:40.396033 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260810 07:28:40.397641 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:40.398754 16148 catalog_manager.cc:1347] Generated new cluster ID: e5bb40fa277840ec89682b29b5f69dfb
I20260810 07:28:40.398833 16148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:40.409682 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:40.401586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50245 (local address 127.0.0.1:33471)
0810 07:28:40.401639 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:40.401642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:40.401882 (+ 240us) server_negotiation.cc:407] Connection header received
0810 07:28:40.401923 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:40.401924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:40.401937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:40.402012 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:40.402685 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:40.403452 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:40.404466 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:40.404812 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:40.404898 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:40.406443 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:40.406445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:40.406445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:40.406455 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:40.408281 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:40.408720 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:40.408721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:40.408722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:40.408796 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:40.409040 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:40.409041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:40.409042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:40.409239 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:40.409373 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:40.409494 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 07:28:40.409572 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:28:40.435128 16148 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:40.435729 16148 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:40.440095 16148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1ce15938c4f4f218b912a950256faf9: Generated new TSK 0
I20260810 07:28:40.440335 16148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:40.447784 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16056
W20260810 07:28:40.455542 16036 connection.cc:535] client connection to 127.0.0.1:33471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33471): Connection reset by peer (error 104)
2026-08-10T07:28:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:40 dist-test-slave-vxj2 krb5kdc[16171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:40 dist-test-slave-vxj2 krb5kdc[16171](info): set up 2 sockets
Aug 10 07:28:40 dist-test-slave-vxj2 krb5kdc[16171](info): commencing operation
krb5kdc: starting...
W20260810 07:28:41.584149 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:41 dist-test-slave-vxj2 krb5kdc[16171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:41.732695 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:41.816738 16188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:41.816974 16188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:41.816998 16188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:41.818194 16188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:41.818241 16188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:41.818256 16188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:41.818269 16188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:41.818280 16188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:41.819796 16188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:41.820029 16188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:41.820221 16188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:41.825506 16194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:41.825764 16188 server_base.cc:1034] running on GCE node
W20260810 07:28:41.825520 16196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:41.825520 16193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:41.826217 16188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:41.826611 16188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:41.827811 16188 hybrid_clock.cc:648] HybridClock initialized: now 1786346921827798 us; error 79 us; skew 500 ppm
Aug 10 07:28:41 dist-test-slave-vxj2 krb5kdc[16171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:41.830578 16188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:41.831032 16188 webserver.cc:460] Webserver started at http://127.0.0.1:43091/ using document root <none> and password file <none>
I20260810 07:28:41.831288 16188 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:41.831338 16188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:41.831482 16188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:41.832629 16188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "047447782b644c7ba294a56c39fbeef9"
format_stamp: "Formatted at 2026-08-10 07:28:41 on dist-test-slave-vxj2"
server_key: "7a30520ad7fcc4ea26d8c976515172a9"
server_key_iv: "6ef81b1c9f726ee81d46c4047601d5ed"
server_key_version: "encryptionkey@0"
I20260810 07:28:41.833024 16188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "047447782b644c7ba294a56c39fbeef9"
format_stamp: "Formatted at 2026-08-10 07:28:41 on dist-test-slave-vxj2"
server_key: "7a30520ad7fcc4ea26d8c976515172a9"
server_key_iv: "6ef81b1c9f726ee81d46c4047601d5ed"
server_key_version: "encryptionkey@0"
I20260810 07:28:41.834513 16188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:41.835321 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:41.835544 16188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:41.835626 16188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "047447782b644c7ba294a56c39fbeef9"
format_stamp: "Formatted at 2026-08-10 07:28:41 on dist-test-slave-vxj2"
server_key: "7a30520ad7fcc4ea26d8c976515172a9"
server_key_iv: "6ef81b1c9f726ee81d46c4047601d5ed"
server_key_version: "encryptionkey@0"
I20260810 07:28:41.835721 16188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:41.879638 16188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:41.881939 16188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:41.882105 16188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:41.886257 16188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37993
I20260810 07:28:41.886322 16279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37993 every 8 connection(s)
I20260810 07:28:41.886744 16188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:41.887714 16281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:41.889282 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16188
I20260810 07:28:41.889427 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:41.889685 24819 external_mini_cluster.cc:1428] Setting key 501a7820fdd6eec00cf2e35c7b7b5883
I20260810 07:28:41.890888 16281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9: Bootstrap starting.
I20260810 07:28:41.891605 16281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:41.891959 16281 log.cc:824] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:41.892719 16281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9: No bootstrap required, opened a new log
I20260810 07:28:41.895238 16281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047447782b644c7ba294a56c39fbeef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260810 07:28:41.895440 16281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:41.895490 16281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047447782b644c7ba294a56c39fbeef9, State: Initialized, Role: FOLLOWER
I20260810 07:28:41.895663 16281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047447782b644c7ba294a56c39fbeef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260810 07:28:41.895740 16281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:41.895769 16281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:41.895838 16281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:41.896281 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:41.890431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36929 (local address 127.0.0.1:37993)
0810 07:28:41.890626 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:41.890639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:41.890673 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:28:41.891230 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:41.891239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:41.891299 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:41.891440 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:41.891743 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:41.893019 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:41.894175 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:41.894600 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:41.894702 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:41.895538 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:41.895547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:41.895548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:41.895819 (+ 271us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:41.895871 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:41.895887 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:41.895977 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:41.896091 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 07:28:41.896117 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260810 07:28:41.896659 16281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047447782b644c7ba294a56c39fbeef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260810 07:28:41.896768 16281 leader_election.cc:302] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047447782b644c7ba294a56c39fbeef9; no voters:
I20260810 07:28:41.896978 16281 leader_election.cc:288] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:41.897012 16286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:41.897207 16286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [term 1 LEADER]: Becoming Leader. State: Replica: 047447782b644c7ba294a56c39fbeef9, State: Running, Role: LEADER
I20260810 07:28:41.897275 16281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:41.897361 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:41.897347 16286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047447782b644c7ba294a56c39fbeef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } }
I20260810 07:28:41.897691 16288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "047447782b644c7ba294a56c39fbeef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047447782b644c7ba294a56c39fbeef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } } }
I20260810 07:28:41.897761 16288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:41.897681 16286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047447782b644c7ba294a56c39fbeef9. Latest consensus state: current_term: 1 leader_uuid: "047447782b644c7ba294a56c39fbeef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047447782b644c7ba294a56c39fbeef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37993 } } }
I20260810 07:28:41.897943 16286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9 [sys.catalog]: This master's current role is: LEADER
W20260810 07:28:41.899425 16311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:28:41.899503 16311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:28:41.899577 16293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:41.899760 16293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:41.901476 16293 catalog_manager.cc:1347] Generated new cluster ID: db35fe7f92c54b0b94073eae2bdff8b4
I20260810 07:28:41.901597 16293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:41.902801 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:41.899034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50015 (local address 127.0.0.1:37993)
0810 07:28:41.899072 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:41.899075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:41.899192 (+ 117us) server_negotiation.cc:407] Connection header received
0810 07:28:41.899264 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:41.899265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:41.899276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:41.899331 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:41.899543 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:41.900296 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:41.901314 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:41.901649 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:41.901727 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:41.902531 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:41.902532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:41.902532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:41.902537 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:41.902552 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:41.902556 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:41.902612 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:41.902716 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 07:28:41.902726 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 07:28:41.918879 16293 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:41.920207 16293 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:41.922777 16293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 047447782b644c7ba294a56c39fbeef9: Generated new TSK 0
I20260810 07:28:41.923053 16293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:41.929524 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16188
W20260810 07:28:41.937662 16169 connection.cc:535] client connection to 127.0.0.1:37993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37993): Connection reset by peer (error 104)
2026-08-10T07:28:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:41 dist-test-slave-vxj2 krb5kdc[16317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:41 dist-test-slave-vxj2 krb5kdc[16317](info): set up 2 sockets
Aug 10 07:28:41 dist-test-slave-vxj2 krb5kdc[16317](info): commencing operation
krb5kdc: starting...
W20260810 07:28:43.090549 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:43 dist-test-slave-vxj2 krb5kdc[16317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:43.229311 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:43.313262 16333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:43.313486 16333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:43.313510 16333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:43.314843 16333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:43.314893 16333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:43.314908 16333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:43.314920 16333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:43.314934 16333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:43.316468 16333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36215
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:43.316742 16333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:43.317019 16333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:43.322458 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:43.322399 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:43.322430 16333 server_base.cc:1034] running on GCE node
W20260810 07:28:43.322400 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:43.323019 16333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:43.323331 16333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:43.324492 16333 hybrid_clock.cc:648] HybridClock initialized: now 1786346923324470 us; error 38 us; skew 500 ppm
Aug 10 07:28:43 dist-test-slave-vxj2 krb5kdc[16317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:43.327219 16333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:43.327760 16333 webserver.cc:460] Webserver started at http://127.0.0.1:34265/ using document root <none> and password file <none>
I20260810 07:28:43.328025 16333 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:43.328101 16333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:43.328261 16333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:43.329491 16333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "2f07f1c85765412fab0c061f1f94d259"
format_stamp: "Formatted at 2026-08-10 07:28:43 on dist-test-slave-vxj2"
server_key: "717b73a54c0a175b7841288ed04f4f4b"
server_key_iv: "3c7100f4354f06cf873885c76372c64b"
server_key_version: "encryptionkey@0"
I20260810 07:28:43.329869 16333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "2f07f1c85765412fab0c061f1f94d259"
format_stamp: "Formatted at 2026-08-10 07:28:43 on dist-test-slave-vxj2"
server_key: "717b73a54c0a175b7841288ed04f4f4b"
server_key_iv: "3c7100f4354f06cf873885c76372c64b"
server_key_version: "encryptionkey@0"
I20260810 07:28:43.331364 16333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:43.332250 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:43.332460 16333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:43.332577 16333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "2f07f1c85765412fab0c061f1f94d259"
format_stamp: "Formatted at 2026-08-10 07:28:43 on dist-test-slave-vxj2"
server_key: "717b73a54c0a175b7841288ed04f4f4b"
server_key_iv: "3c7100f4354f06cf873885c76372c64b"
server_key_version: "encryptionkey@0"
I20260810 07:28:43.332657 16333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:43.336455 16333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:43.338871 16333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:43.339053 16333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:43.343487 16333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36215
I20260810 07:28:43.343530 16416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36215 every 8 connection(s)
I20260810 07:28:43.343983 16333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:43.345103 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16333
I20260810 07:28:43.345105 16417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:43.345269 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:43.345495 24819 external_mini_cluster.cc:1428] Setting key 5b51598f66203d71526b02a4fa656561
I20260810 07:28:43.348363 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:43.346169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45535 (local address 127.0.0.1:36215)
0810 07:28:43.346377 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:43.346386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:43.346415 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:28:43.346948 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:43.346957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:43.346987 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:43.347124 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:43.347827 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:43.347831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:43.347831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:43.347835 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:43.347872 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:43.347883 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:43.347986 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:43.348088 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 07:28:43.348153 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260810 07:28:43.349320 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259: Bootstrap starting.
I20260810 07:28:43.349731 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:43.350090 16417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:43.350411 16417 log.cc:824] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:43.351315 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259: No bootstrap required, opened a new log
I20260810 07:28:43.352488 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:43.351284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55417 (local address 127.0.0.1:36215)
0810 07:28:43.351351 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:43.351353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:43.351484 (+ 131us) server_negotiation.cc:407] Connection header received
0810 07:28:43.351550 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:43.351551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:43.351556 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:43.351602 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:43.352224 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:43.352225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:43.352225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:43.352229 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:43.352242 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:43.352246 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:43.352291 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:43.352371 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 07:28:43.352383 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:28:43.353922 16417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f07f1c85765412fab0c061f1f94d259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260810 07:28:43.354092 16417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:43.354140 16417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f07f1c85765412fab0c061f1f94d259, State: Initialized, Role: FOLLOWER
I20260810 07:28:43.354276 16417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f07f1c85765412fab0c061f1f94d259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260810 07:28:43.354383 16417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:43.354429 16417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:43.354482 16417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:43.355111 16417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f07f1c85765412fab0c061f1f94d259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260810 07:28:43.355239 16417 leader_election.cc:302] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f07f1c85765412fab0c061f1f94d259; no voters:
I20260810 07:28:43.355436 16417 leader_election.cc:288] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:43.355506 16429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:43.355700 16417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:43.355731 16429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [term 1 LEADER]: Becoming Leader. State: Replica: 2f07f1c85765412fab0c061f1f94d259, State: Running, Role: LEADER
I20260810 07:28:43.355830 16429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f07f1c85765412fab0c061f1f94d259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260810 07:28:43.356236 16430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f07f1c85765412fab0c061f1f94d259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f07f1c85765412fab0c061f1f94d259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } } }
I20260810 07:28:43.356238 16431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f07f1c85765412fab0c061f1f94d259. Latest consensus state: current_term: 1 leader_uuid: "2f07f1c85765412fab0c061f1f94d259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f07f1c85765412fab0c061f1f94d259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } } }
I20260810 07:28:43.356391 16430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:43.356407 16431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:43.356664 16439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:43.357306 16439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:43.359071 16439 catalog_manager.cc:1347] Generated new cluster ID: 2332cc6f4dec4f00b1e5fcc7a1c5f1c8
I20260810 07:28:43.359129 16439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:43.366197 16439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:43.366439 16439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:43.371007 16439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f07f1c85765412fab0c061f1f94d259: Generated new TSK 0
I20260810 07:28:43.371291 16439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:43.377487 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16333
2026-08-10T07:28:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:43 dist-test-slave-vxj2 krb5kdc[16454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:43 dist-test-slave-vxj2 krb5kdc[16454](info): set up 2 sockets
Aug 10 07:28:43 dist-test-slave-vxj2 krb5kdc[16454](info): commencing operation
krb5kdc: starting...
W20260810 07:28:44.508685 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:44 dist-test-slave-vxj2 krb5kdc[16454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:44.644055 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:44.731230 16468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:44.731513 16468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:44.731556 16468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:44.732769 16468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:44.732884 16468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:44.732914 16468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:44.732944 16468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:44.732975 16468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:44.734620 16468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:44.734961 16468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:44.735224 16468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:44.740470 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:44.740507 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:44.740470 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:44.740694 16468 server_base.cc:1034] running on GCE node
I20260810 07:28:44.741096 16468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:44.741454 16468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:44.742645 16468 hybrid_clock.cc:648] HybridClock initialized: now 1786346924742639 us; error 41 us; skew 500 ppm
Aug 10 07:28:44 dist-test-slave-vxj2 krb5kdc[16454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:44.745615 16468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:44.746065 16468 webserver.cc:460] Webserver started at http://127.0.0.1:44003/ using document root <none> and password file <none>
I20260810 07:28:44.746408 16468 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:44.746460 16468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:44.746636 16468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:44.747844 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "e9d96ec82ba4432fbda1c48c70c0cb87"
format_stamp: "Formatted at 2026-08-10 07:28:44 on dist-test-slave-vxj2"
server_key: "4ede62e2a71eb4ebdf0ee3b2966fa685"
server_key_iv: "77017106abea2de954e41e919d96ddfd"
server_key_version: "encryptionkey@0"
I20260810 07:28:44.748222 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "e9d96ec82ba4432fbda1c48c70c0cb87"
format_stamp: "Formatted at 2026-08-10 07:28:44 on dist-test-slave-vxj2"
server_key: "4ede62e2a71eb4ebdf0ee3b2966fa685"
server_key_iv: "77017106abea2de954e41e919d96ddfd"
server_key_version: "encryptionkey@0"
I20260810 07:28:44.749914 16468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:44.750798 16487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:44.751048 16468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:44.751157 16468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "e9d96ec82ba4432fbda1c48c70c0cb87"
format_stamp: "Formatted at 2026-08-10 07:28:44 on dist-test-slave-vxj2"
server_key: "4ede62e2a71eb4ebdf0ee3b2966fa685"
server_key_iv: "77017106abea2de954e41e919d96ddfd"
server_key_version: "encryptionkey@0"
I20260810 07:28:44.751245 16468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:44.763940 16468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:44.766496 16468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:44.766698 16468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:44.771206 16468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41999
I20260810 07:28:44.771243 16549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41999 every 8 connection(s)
I20260810 07:28:44.771621 16468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:44.772543 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:44.774974 16550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87: Bootstrap starting.
I20260810 07:28:44.775542 16550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:44.775777 16550 log.cc:824] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:44.776526 16550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87: No bootstrap required, opened a new log
I20260810 07:28:44.778194 16550 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41999 } }
I20260810 07:28:44.778337 16550 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:44.778369 16550 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d96ec82ba4432fbda1c48c70c0cb87, State: Initialized, Role: FOLLOWER
I20260810 07:28:44.778512 16550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41999 } }
I20260810 07:28:44.778601 16550 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:44.778625 16550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:44.778654 16550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:44.779157 16550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41999 } }
I20260810 07:28:44.779248 16550 leader_election.cc:302] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d96ec82ba4432fbda1c48c70c0cb87; no voters:
I20260810 07:28:44.779426 16550 leader_election.cc:288] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:44.779458 16553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:44.779583 16553 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [term 1 LEADER]: Becoming Leader. State: Replica: e9d96ec82ba4432fbda1c48c70c0cb87, State: Running, Role: LEADER
I20260810 07:28:44.779726 16550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:44.779742 16553 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41999 } }
I20260810 07:28:44.780316 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16468
I20260810 07:28:44.780309 16554 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41999 } } }
I20260810 07:28:44.780403 16554 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:44.780457 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:44.780624 16563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:44.780665 24819 external_mini_cluster.cc:1428] Setting key 64f448c88d349ec1f524c998bc458caf
I20260810 07:28:44.781196 16563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:44.781375 16556 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d96ec82ba4432fbda1c48c70c0cb87. Latest consensus state: current_term: 1 leader_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d96ec82ba4432fbda1c48c70c0cb87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41999 } } }
I20260810 07:28:44.781545 16556 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:44.782987 16563 catalog_manager.cc:1347] Generated new cluster ID: 3024077927fc4d1eac691ac688567392
I20260810 07:28:44.783048 16563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:28:44 dist-test-slave-vxj2 krb5kdc[16454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:28:44.791764 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:44.781542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40537 (local address 127.0.0.1:41999)
0810 07:28:44.781947 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:44.781957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:44.781990 (+ 33us) server_negotiation.cc:407] Connection header received
0810 07:28:44.782113 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:44.782120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:44.782177 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:44.783293 (+ 1116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:44.783379 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:44.784304 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:44.785137 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:44.785602 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:44.785728 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:44.788180 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:44.788184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:44.788185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:44.788198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:44.790023 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:44.790443 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:44.790447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:44.790447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:44.790564 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:44.790789 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:44.790789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:44.790790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:44.791084 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:44.791265 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:44.791388 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 07:28:44.791541 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":100,"threads_started":1}
I20260810 07:28:44.792733 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:44.799906 16563 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:44.800678 16563 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:44.800813 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:44.794239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45895 (local address 127.0.0.1:41999)
0810 07:28:44.794278 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:44.794280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:44.794414 (+ 134us) server_negotiation.cc:407] Connection header received
0810 07:28:44.794489 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:44.794490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:44.794510 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:44.794559 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:44.795116 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:44.795641 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:44.796383 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:44.796608 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:44.796684 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:44.797914 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:44.797915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:44.797916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:44.797922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:44.799448 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:44.799852 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:44.799853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:44.799853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:44.799914 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:44.800191 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:44.800192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:44.800192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:44.800304 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:44.800373 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:44.800584 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 07:28:44.800669 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 07:28:44.810539 16563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d96ec82ba4432fbda1c48c70c0cb87: Generated new TSK 0
I20260810 07:28:44.810796 16563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:44.818334 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16468
2026-08-10T07:28:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:44 dist-test-slave-vxj2 krb5kdc[16585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:44 dist-test-slave-vxj2 krb5kdc[16585](info): set up 2 sockets
Aug 10 07:28:44 dist-test-slave-vxj2 krb5kdc[16585](info): commencing operation
krb5kdc: starting...
W20260810 07:28:45.944336 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:46 dist-test-slave-vxj2 krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:46.080641 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:46.168341 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:46.168593 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:46.168620 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:46.170317 16600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:46.170394 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:46.170413 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:46.170428 16600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:46.170442 16600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:46.172616 16600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:46.172956 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:46.173256 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:46.179064 16609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:46.179083 16606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:46.179060 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:46.179302 16600 server_base.cc:1034] running on GCE node
I20260810 07:28:46.179736 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:46.180040 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:46.181236 16600 hybrid_clock.cc:648] HybridClock initialized: now 1786346926181234 us; error 49 us; skew 500 ppm
Aug 10 07:28:46 dist-test-slave-vxj2 krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:46.187361 16600 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:46.187983 16600 webserver.cc:460] Webserver started at http://127.0.0.1:38585/ using document root <none> and password file <none>
I20260810 07:28:46.188275 16600 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:46.188364 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:46.188535 16600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:46.189889 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "6b255e2e9e4f4fb6870a1677a6718047"
format_stamp: "Formatted at 2026-08-10 07:28:46 on dist-test-slave-vxj2"
server_key: "a740ef3e0938e8b07c936fd78158bdae"
server_key_iv: "7a9dd15a2afe6af45aafeeddaac10647"
server_key_version: "encryptionkey@0"
I20260810 07:28:46.190311 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "6b255e2e9e4f4fb6870a1677a6718047"
format_stamp: "Formatted at 2026-08-10 07:28:46 on dist-test-slave-vxj2"
server_key: "a740ef3e0938e8b07c936fd78158bdae"
server_key_iv: "7a9dd15a2afe6af45aafeeddaac10647"
server_key_version: "encryptionkey@0"
I20260810 07:28:46.191869 16600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:46.192716 16618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:46.192950 16600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:46.193073 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "6b255e2e9e4f4fb6870a1677a6718047"
format_stamp: "Formatted at 2026-08-10 07:28:46 on dist-test-slave-vxj2"
server_key: "a740ef3e0938e8b07c936fd78158bdae"
server_key_iv: "7a9dd15a2afe6af45aafeeddaac10647"
server_key_version: "encryptionkey@0"
I20260810 07:28:46.193159 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:46.202025 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:46.204926 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:46.205127 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:46.210095 16600 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36579
I20260810 07:28:46.210152 16680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36579 every 8 connection(s)
I20260810 07:28:46.210594 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:46.211558 16681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:46.214186 16681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047: Bootstrap starting.
I20260810 07:28:46.214862 16681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:46.215175 16681 log.cc:824] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:46.216032 16681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047: No bootstrap required, opened a new log
I20260810 07:28:46.217473 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16600
I20260810 07:28:46.217644 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:46.217954 24819 external_mini_cluster.cc:1428] Setting key 8d6ac5142312c29a56b945fdab729784
I20260810 07:28:46.217978 16681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b255e2e9e4f4fb6870a1677a6718047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36579 } }
I20260810 07:28:46.218191 16681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:46.218250 16681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b255e2e9e4f4fb6870a1677a6718047, State: Initialized, Role: FOLLOWER
I20260810 07:28:46.218454 16681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b255e2e9e4f4fb6870a1677a6718047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36579 } }
I20260810 07:28:46.218554 16681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:46.218617 16681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:46.218676 16681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:46.219525 16681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b255e2e9e4f4fb6870a1677a6718047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36579 } }
I20260810 07:28:46.219650 16681 leader_election.cc:302] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b255e2e9e4f4fb6870a1677a6718047; no voters:
I20260810 07:28:46.219846 16681 leader_election.cc:288] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:46.219882 16689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:46.219993 16689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [term 1 LEADER]: Becoming Leader. State: Replica: 6b255e2e9e4f4fb6870a1677a6718047, State: Running, Role: LEADER
I20260810 07:28:46.220137 16681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:46.220136 16689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b255e2e9e4f4fb6870a1677a6718047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36579 } }
I20260810 07:28:46.220633 16690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b255e2e9e4f4fb6870a1677a6718047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b255e2e9e4f4fb6870a1677a6718047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36579 } } }
I20260810 07:28:46.220857 16690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:46.220798 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b255e2e9e4f4fb6870a1677a6718047. Latest consensus state: current_term: 1 leader_uuid: "6b255e2e9e4f4fb6870a1677a6718047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b255e2e9e4f4fb6870a1677a6718047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36579 } } }
I20260810 07:28:46.221127 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:46.221592 16699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:46.222124 16699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:46.223971 16699 catalog_manager.cc:1347] Generated new cluster ID: e2b82f7974be4b84be0ebb7288d0eaba
I20260810 07:28:46.224028 16699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:46.225546 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:46.218607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46343 (local address 127.0.0.1:36579)
0810 07:28:46.218840 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:46.218851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:46.218942 (+ 91us) server_negotiation.cc:407] Connection header received
0810 07:28:46.219418 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:46.219429 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:46.219503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:46.219684 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:46.220022 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:46.221553 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:46.222839 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:46.223541 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:46.223685 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:46.224621 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:46.224626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:46.224626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:46.224881 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:46.224923 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:46.224941 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:46.224991 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:46.225212 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:28:46.225303 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":114,"threads_started":1}
I20260810 07:28:46.230480 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:46.236176 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:46.232103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38967 (local address 127.0.0.1:36579)
0810 07:28:46.232150 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:46.232152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:46.232328 (+ 176us) server_negotiation.cc:407] Connection header received
0810 07:28:46.232367 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:46.232368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:46.232382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:46.232455 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:46.232615 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:46.233484 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:46.234551 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:46.234966 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:46.235057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:46.235860 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:46.235862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:46.235862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:46.235884 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:46.235905 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:46.235910 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:46.235977 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:46.236036 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 07:28:46.236073 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:28:46.251389 16699 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:46.252031 16699 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:46.255133 16699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b255e2e9e4f4fb6870a1677a6718047: Generated new TSK 0
I20260810 07:28:46.255395 16699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:46.261013 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16600
2026-08-10T07:28:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:46 dist-test-slave-vxj2 krb5kdc[16720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:46 dist-test-slave-vxj2 krb5kdc[16720](info): set up 2 sockets
Aug 10 07:28:46 dist-test-slave-vxj2 krb5kdc[16720](info): commencing operation
krb5kdc: starting...
W20260810 07:28:47.402925 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:47 dist-test-slave-vxj2 krb5kdc[16720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:47.540798 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:47.625568 16734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:47.625803 16734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:47.625830 16734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:47.627123 16734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:47.627169 16734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:47.627184 16734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:47.627197 16734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:47.627209 16734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:47.628679 16734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:47.628896 16734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:47.629088 16734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:47.634222 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:47.634270 16734 server_base.cc:1034] running on GCE node
W20260810 07:28:47.634209 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:47.634225 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:47.634840 16734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:47.635169 16734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:47.636370 16734 hybrid_clock.cc:648] HybridClock initialized: now 1786346927636341 us; error 45 us; skew 500 ppm
Aug 10 07:28:47 dist-test-slave-vxj2 krb5kdc[16720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:47.639145 16734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:47.639663 16734 webserver.cc:460] Webserver started at http://127.0.0.1:40097/ using document root <none> and password file <none>
I20260810 07:28:47.639986 16734 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:47.640069 16734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:47.640229 16734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:47.641439 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "98bb0d11f504460e9bd02d19eeebac27"
format_stamp: "Formatted at 2026-08-10 07:28:47 on dist-test-slave-vxj2"
server_key: "f13a060c0e36fbac3bcd05535dde0ef9"
server_key_iv: "ff343a108ae404cf4dc4568797e928ed"
server_key_version: "encryptionkey@0"
I20260810 07:28:47.641815 16734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "98bb0d11f504460e9bd02d19eeebac27"
format_stamp: "Formatted at 2026-08-10 07:28:47 on dist-test-slave-vxj2"
server_key: "f13a060c0e36fbac3bcd05535dde0ef9"
server_key_iv: "ff343a108ae404cf4dc4568797e928ed"
server_key_version: "encryptionkey@0"
I20260810 07:28:47.643275 16734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:47.644208 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:47.644440 16734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:47.644557 16734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "98bb0d11f504460e9bd02d19eeebac27"
format_stamp: "Formatted at 2026-08-10 07:28:47 on dist-test-slave-vxj2"
server_key: "f13a060c0e36fbac3bcd05535dde0ef9"
server_key_iv: "ff343a108ae404cf4dc4568797e928ed"
server_key_version: "encryptionkey@0"
I20260810 07:28:47.644639 16734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:47.648233 16734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:47.650522 16734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:47.650682 16734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:47.654861 16734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45375
I20260810 07:28:47.654922 16811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45375 every 8 connection(s)
I20260810 07:28:47.655287 16734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:47.656313 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16734
I20260810 07:28:47.656410 16812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:47.656479 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:47.656684 24819 external_mini_cluster.cc:1428] Setting key db102c26241cd18611e72f7977f424d3
I20260810 07:28:47.659226 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:47.657288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48279 (local address 127.0.0.1:45375)
0810 07:28:47.657478 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:47.657486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:47.657514 (+ 28us) server_negotiation.cc:407] Connection header received
0810 07:28:47.657960 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:47.657967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:47.657995 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:47.658114 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:47.658779 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:47.658783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:47.658783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:47.658787 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:47.658823 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:47.658834 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:47.658891 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:47.658969 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 07:28:47.659027 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260810 07:28:47.660018 16812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27: Bootstrap starting.
I20260810 07:28:47.660184 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:47.660704 16812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:47.661007 16812 log.cc:824] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:47.661870 16812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27: No bootstrap required, opened a new log
I20260810 07:28:47.662889 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:47.661749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45741 (local address 127.0.0.1:45375)
0810 07:28:47.661793 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:47.661795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:47.661966 (+ 171us) server_negotiation.cc:407] Connection header received
0810 07:28:47.662029 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:47.662029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:47.662034 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:47.662094 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:47.662663 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:47.662663 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:47.662664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:47.662667 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:47.662681 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:47.662685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:47.662723 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:47.662804 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 07:28:47.662815 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:28:47.664186 16812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bb0d11f504460e9bd02d19eeebac27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45375 } }
I20260810 07:28:47.664333 16812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:47.664357 16812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bb0d11f504460e9bd02d19eeebac27, State: Initialized, Role: FOLLOWER
I20260810 07:28:47.664502 16812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bb0d11f504460e9bd02d19eeebac27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45375 } }
I20260810 07:28:47.664592 16812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:47.664636 16812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:47.664688 16812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:47.665354 16812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bb0d11f504460e9bd02d19eeebac27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45375 } }
I20260810 07:28:47.665472 16812 leader_election.cc:302] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98bb0d11f504460e9bd02d19eeebac27; no voters:
I20260810 07:28:47.665678 16812 leader_election.cc:288] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:47.665750 16826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:47.665943 16812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:47.665982 16826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [term 1 LEADER]: Becoming Leader. State: Replica: 98bb0d11f504460e9bd02d19eeebac27, State: Running, Role: LEADER
I20260810 07:28:47.666076 16826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bb0d11f504460e9bd02d19eeebac27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45375 } }
I20260810 07:28:47.666703 16829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98bb0d11f504460e9bd02d19eeebac27. Latest consensus state: current_term: 1 leader_uuid: "98bb0d11f504460e9bd02d19eeebac27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bb0d11f504460e9bd02d19eeebac27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45375 } } }
I20260810 07:28:47.666833 16829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:47.666710 16828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98bb0d11f504460e9bd02d19eeebac27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bb0d11f504460e9bd02d19eeebac27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45375 } } }
I20260810 07:28:47.666999 16828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:47.667150 16839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:47.667667 16839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:47.669230 16839 catalog_manager.cc:1347] Generated new cluster ID: 58452018bf664661911d52fc435d7525
I20260810 07:28:47.669286 16839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:47.676542 16839 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:47.676749 16839 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:47.681072 16839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98bb0d11f504460e9bd02d19eeebac27: Generated new TSK 0
I20260810 07:28:47.681398 16839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:47.683959 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16734
2026-08-10T07:28:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:47 dist-test-slave-vxj2 krb5kdc[16852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:47 dist-test-slave-vxj2 krb5kdc[16852](info): set up 2 sockets
Aug 10 07:28:47 dist-test-slave-vxj2 krb5kdc[16852](info): commencing operation
krb5kdc: starting...
W20260810 07:28:48.815652 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:48 dist-test-slave-vxj2 krb5kdc[16852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:48.944681 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:49.031491 16867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:49.031778 16867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:49.031831 16867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:49.033041 16867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:49.033125 16867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:49.033196 16867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:49.033228 16867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:49.033248 16867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:49.034821 16867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:49.035122 16867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:49.035364 16867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:49.040663 16875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:49.040632 16872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:49.040678 16867 server_base.cc:1034] running on GCE node
W20260810 07:28:49.040648 16873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:49.041306 16867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:49.041615 16867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:49.042825 16867 hybrid_clock.cc:648] HybridClock initialized: now 1786346929042793 us; error 41 us; skew 500 ppm
Aug 10 07:28:49 dist-test-slave-vxj2 krb5kdc[16852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:49.045825 16867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:49.046310 16867 webserver.cc:460] Webserver started at http://127.0.0.1:44203/ using document root <none> and password file <none>
I20260810 07:28:49.046578 16867 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:49.046653 16867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:49.046806 16867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:49.047940 16867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "6ad6c581fb21436eaceeb123baa76090"
format_stamp: "Formatted at 2026-08-10 07:28:49 on dist-test-slave-vxj2"
server_key: "54271c8d2cdc40d4c6b3d4808171c45a"
server_key_iv: "e7d199f981e7940bad5f6b8e6e87f46c"
server_key_version: "encryptionkey@0"
I20260810 07:28:49.048332 16867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "6ad6c581fb21436eaceeb123baa76090"
format_stamp: "Formatted at 2026-08-10 07:28:49 on dist-test-slave-vxj2"
server_key: "54271c8d2cdc40d4c6b3d4808171c45a"
server_key_iv: "e7d199f981e7940bad5f6b8e6e87f46c"
server_key_version: "encryptionkey@0"
I20260810 07:28:49.049854 16867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:49.050642 16884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:49.050832 16867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:49.050951 16867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "6ad6c581fb21436eaceeb123baa76090"
format_stamp: "Formatted at 2026-08-10 07:28:49 on dist-test-slave-vxj2"
server_key: "54271c8d2cdc40d4c6b3d4808171c45a"
server_key_iv: "e7d199f981e7940bad5f6b8e6e87f46c"
server_key_version: "encryptionkey@0"
I20260810 07:28:49.051028 16867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:49.075558 16867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:49.077963 16867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:49.078152 16867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:49.082556 16867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45525
I20260810 07:28:49.082613 16940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45525 every 8 connection(s)
I20260810 07:28:49.082998 16867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:49.083959 16941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:49.086431 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090: Bootstrap starting.
I20260810 07:28:49.087088 16941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:49.087389 16941 log.cc:824] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:49.088207 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090: No bootstrap required, opened a new log
I20260810 07:28:49.089928 16941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad6c581fb21436eaceeb123baa76090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260810 07:28:49.090075 16941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:49.090097 16941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad6c581fb21436eaceeb123baa76090, State: Initialized, Role: FOLLOWER
I20260810 07:28:49.090260 16941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad6c581fb21436eaceeb123baa76090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260810 07:28:49.090328 16941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:49.090350 16941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:49.090416 16941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:49.090685 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16867
I20260810 07:28:49.090826 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:49.091049 24819 external_mini_cluster.cc:1428] Setting key 7e0d36a706f66afeec99feaaab5bee70
I20260810 07:28:49.091100 16941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad6c581fb21436eaceeb123baa76090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260810 07:28:49.091219 16941 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad6c581fb21436eaceeb123baa76090; no voters:
I20260810 07:28:49.091431 16941 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:49.091495 16944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:49.091670 16944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad6c581fb21436eaceeb123baa76090, State: Running, Role: LEADER
I20260810 07:28:49.091745 16941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:49.091768 16944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad6c581fb21436eaceeb123baa76090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } }
I20260810 07:28:49.092293 16950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad6c581fb21436eaceeb123baa76090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad6c581fb21436eaceeb123baa76090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } } }
I20260810 07:28:49.092387 16950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:49.092594 16952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:49.093257 16952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:49.093217 16946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad6c581fb21436eaceeb123baa76090. Latest consensus state: current_term: 1 leader_uuid: "6ad6c581fb21436eaceeb123baa76090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad6c581fb21436eaceeb123baa76090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45525 } } }
I20260810 07:28:49.093737 16946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:49.095166 16952 catalog_manager.cc:1347] Generated new cluster ID: dcd41d5b93b34e33b2d20892336e08e7
I20260810 07:28:49.095238 16952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:28:49 dist-test-slave-vxj2 krb5kdc[16852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:28:49.101468 16952 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:49.102075 16952 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:49.103647 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:49.091558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:48789 (local address 127.0.0.1:45525)
0810 07:28:49.092231 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:49.092242 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:49.092281 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:28:49.092404 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:49.092411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:49.092473 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:49.093663 (+ 1190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:49.093760 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:49.094841 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:49.095931 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:49.096541 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:49.096637 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:49.099606 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:49.099610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:49.099611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:49.099625 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:49.101833 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:49.102321 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:49.102325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:49.102325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:49.102451 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:49.102705 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:49.102706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:49.102707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:49.103020 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:49.103204 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:49.103297 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 07:28:49.103418 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":62,"threads_started":1}
I20260810 07:28:49.104679 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:49.108937 16952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ad6c581fb21436eaceeb123baa76090: Generated new TSK 0
I20260810 07:28:49.109138 16952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:49.113744 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:49.106245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:45889 (local address 127.0.0.1:45525)
0810 07:28:49.106302 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:49.106304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:49.106410 (+ 106us) server_negotiation.cc:407] Connection header received
0810 07:28:49.106473 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:49.106474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:49.106485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:49.106554 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:49.107206 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:49.107978 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:49.109121 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:49.109475 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:49.109534 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:49.110981 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:49.110982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:49.110982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:49.110989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:49.112539 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:49.112961 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:49.112962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:49.112962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:49.113014 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:49.113278 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:49.113279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:49.113279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:49.113385 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:49.113462 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:49.113629 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 07:28:49.113682 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:28:49.115983 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16867
W20260810 07:28:49.123582 16849 connection.cc:535] client connection to 127.0.0.1:45525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45525): Connection reset by peer (error 104)
2026-08-10T07:28:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:49 dist-test-slave-vxj2 krb5kdc[16977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:49 dist-test-slave-vxj2 krb5kdc[16977](info): set up 2 sockets
Aug 10 07:28:49 dist-test-slave-vxj2 krb5kdc[16977](info): commencing operation
krb5kdc: starting...
W20260810 07:28:50.235531 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:50 dist-test-slave-vxj2 krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:50.368247 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:50.454519 16993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:50.454744 16993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:50.454769 16993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:50.455971 16993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:50.456017 16993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:50.456032 16993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:50.456044 16993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:50.456056 16993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:50.457580 16993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:50.457796 16993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:50.457981 16993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:50.463042 17001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:50.463189 16993 server_base.cc:1034] running on GCE node
W20260810 07:28:50.463056 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:50.463064 17003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:50.463568 16993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:50.463872 16993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:50.465019 16993 hybrid_clock.cc:648] HybridClock initialized: now 1786346930464997 us; error 39 us; skew 500 ppm
Aug 10 07:28:50 dist-test-slave-vxj2 krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:50.467767 16993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:50.468315 16993 webserver.cc:460] Webserver started at http://127.0.0.1:41997/ using document root <none> and password file <none>
I20260810 07:28:50.468552 16993 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:50.468621 16993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:50.468780 16993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:50.469954 16993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb"
format_stamp: "Formatted at 2026-08-10 07:28:50 on dist-test-slave-vxj2"
server_key: "dc8e25e092416d857dcebcc00817d723"
server_key_iv: "8ac1cd0f7624513cfac1bace1b8dde18"
server_key_version: "encryptionkey@0"
I20260810 07:28:50.470332 16993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb"
format_stamp: "Formatted at 2026-08-10 07:28:50 on dist-test-slave-vxj2"
server_key: "dc8e25e092416d857dcebcc00817d723"
server_key_iv: "8ac1cd0f7624513cfac1bace1b8dde18"
server_key_version: "encryptionkey@0"
I20260810 07:28:50.471737 16993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:28:50.472599 17011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:50.472862 16993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:28:50.472986 16993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb"
format_stamp: "Formatted at 2026-08-10 07:28:50 on dist-test-slave-vxj2"
server_key: "dc8e25e092416d857dcebcc00817d723"
server_key_iv: "8ac1cd0f7624513cfac1bace1b8dde18"
server_key_version: "encryptionkey@0"
I20260810 07:28:50.473066 16993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:50.487349 16993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:50.489881 16993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:50.490067 16993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:50.494645 16993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39711
I20260810 07:28:50.494684 17070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39711 every 8 connection(s)
I20260810 07:28:50.495060 16993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:50.495986 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:50.498462 17071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb: Bootstrap starting.
I20260810 07:28:50.499123 17071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:50.499423 17071 log.cc:824] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:50.500269 17071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb: No bootstrap required, opened a new log
I20260810 07:28:50.502065 17071 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260810 07:28:50.502211 17071 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:50.502234 17071 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a4d1e944124dc68089ff3eb6e4ddeb, State: Initialized, Role: FOLLOWER
I20260810 07:28:50.502409 17071 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260810 07:28:50.502473 17071 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:50.502525 17071 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:50.502581 17071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:50.503170 17071 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260810 07:28:50.503288 17071 leader_election.cc:302] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a4d1e944124dc68089ff3eb6e4ddeb; no voters:
I20260810 07:28:50.503481 17071 leader_election.cc:288] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:50.503539 17075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:50.503762 17075 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [term 1 LEADER]: Becoming Leader. State: Replica: d6a4d1e944124dc68089ff3eb6e4ddeb, State: Running, Role: LEADER
I20260810 07:28:50.503775 17071 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:50.503880 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 16993
I20260810 07:28:50.503880 17075 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260810 07:28:50.504004 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:50.504243 24819 external_mini_cluster.cc:1428] Setting key f6a40fcab86b47af57e496ea223dfd09
I20260810 07:28:50.504261 17077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a4d1e944124dc68089ff3eb6e4ddeb. Latest consensus state: current_term: 1 leader_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } } }
I20260810 07:28:50.504350 17077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:50.504513 17076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a4d1e944124dc68089ff3eb6e4ddeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } } }
I20260810 07:28:50.504643 17076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:50.505295 17079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:50.505839 17079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:50.507606 17079 catalog_manager.cc:1347] Generated new cluster ID: a0689e8042fd45c6a6b9db63047ceebd
I20260810 07:28:50.507712 17079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:50.511415 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:50.504986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:54695 (local address 127.0.0.1:39711)
0810 07:28:50.505256 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:50.505265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:50.505307 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:28:50.505442 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:50.505449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:50.505513 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:50.505674 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:50.505919 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:50.507307 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:50.508507 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:50.509254 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:50.510263 (+ 1009us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:50.510446 (+ 183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:50.510449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:50.510450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:50.510678 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:50.510714 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:50.510728 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:50.511093 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:50.511140 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 07:28:50.511170 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260810 07:28:50.512504 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:50.513060 17079 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:50.513595 17079 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:50.516474 17079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a4d1e944124dc68089ff3eb6e4ddeb: Generated new TSK 0
I20260810 07:28:50.516655 17079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:50.517843 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:50.514174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:55591 (local address 127.0.0.1:39711)
0810 07:28:50.514238 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:50.514240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:50.514384 (+ 144us) server_negotiation.cc:407] Connection header received
0810 07:28:50.514452 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:50.514453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:50.514464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:50.514530 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:50.514705 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:50.515466 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:50.516636 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:50.516915 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:50.516974 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:50.517637 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:50.517638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:50.517638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:50.517642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:50.517656 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:50.517660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:50.517711 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:50.517774 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 07:28:50.517785 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:28:50.519735 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 16993
W20260810 07:28:50.529068 16975 connection.cc:535] client connection to 127.0.0.1:39711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39711): Connection reset by peer (error 104)
2026-08-10T07:28:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:50 dist-test-slave-vxj2 krb5kdc[17105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:50 dist-test-slave-vxj2 krb5kdc[17105](info): set up 2 sockets
Aug 10 07:28:50 dist-test-slave-vxj2 krb5kdc[17105](info): commencing operation
krb5kdc: starting...
W20260810 07:28:51.645097 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:51 dist-test-slave-vxj2 krb5kdc[17105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:51.793563 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:51.876535 17119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:51.876824 17119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:51.876863 17119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:51.878054 17119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:51.878131 17119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:51.878154 17119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:51.878176 17119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:51.878204 17119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:51.879703 17119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:51.880013 17119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:51.880255 17119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:51.885418 17126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:51.885393 17129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:51.885388 17125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:51.885618 17119 server_base.cc:1034] running on GCE node
I20260810 07:28:51.885908 17119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:51.886201 17119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:51.887353 17119 hybrid_clock.cc:648] HybridClock initialized: now 1786346931887329 us; error 42 us; skew 500 ppm
Aug 10 07:28:51 dist-test-slave-vxj2 krb5kdc[17105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:51.890161 17119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:51.890681 17119 webserver.cc:460] Webserver started at http://127.0.0.1:39743/ using document root <none> and password file <none>
I20260810 07:28:51.890923 17119 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:51.890974 17119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:51.891106 17119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:51.892262 17119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "569f6746234344ccae2caf52c68ccb8e"
format_stamp: "Formatted at 2026-08-10 07:28:51 on dist-test-slave-vxj2"
server_key: "2f00dce66e9e8b7c4f14e47e187624c5"
server_key_iv: "221d6314050bd3cdd9ee992540393dd2"
server_key_version: "encryptionkey@0"
I20260810 07:28:51.892647 17119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "569f6746234344ccae2caf52c68ccb8e"
format_stamp: "Formatted at 2026-08-10 07:28:51 on dist-test-slave-vxj2"
server_key: "2f00dce66e9e8b7c4f14e47e187624c5"
server_key_iv: "221d6314050bd3cdd9ee992540393dd2"
server_key_version: "encryptionkey@0"
I20260810 07:28:51.894178 17119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:28:51.895040 17137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:51.895231 17119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:51.895339 17119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "569f6746234344ccae2caf52c68ccb8e"
format_stamp: "Formatted at 2026-08-10 07:28:51 on dist-test-slave-vxj2"
server_key: "2f00dce66e9e8b7c4f14e47e187624c5"
server_key_iv: "221d6314050bd3cdd9ee992540393dd2"
server_key_version: "encryptionkey@0"
I20260810 07:28:51.895418 17119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:51.898974 17119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:51.900960 17119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:51.901119 17119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:51.905341 17119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34421
I20260810 07:28:51.905417 17196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34421 every 8 connection(s)
I20260810 07:28:51.905776 17119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:51.906888 17197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:51.909138 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17119
I20260810 07:28:51.909301 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:51.909523 24819 external_mini_cluster.cc:1428] Setting key 052af6cc44b4a156653ece54325c0eef
I20260810 07:28:51.909606 17197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e: Bootstrap starting.
I20260810 07:28:51.910406 17197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:51.910735 17197 log.cc:824] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:51.911564 17197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e: No bootstrap required, opened a new log
I20260810 07:28:51.912387 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:51.910079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:45219 (local address 127.0.0.1:34421)
0810 07:28:51.910308 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:51.910335 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:51.910380 (+ 45us) server_negotiation.cc:407] Connection header received
0810 07:28:51.911046 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:51.911054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:51.911091 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:51.911262 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:51.911925 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:51.911929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:51.911930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:51.911934 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:51.911984 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:51.912001 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:51.912099 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:51.912133 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 07:28:51.912180 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
I20260810 07:28:51.913508 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:51.914208 17197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569f6746234344ccae2caf52c68ccb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260810 07:28:51.914362 17197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:51.914386 17197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569f6746234344ccae2caf52c68ccb8e, State: Initialized, Role: FOLLOWER
I20260810 07:28:51.914547 17197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569f6746234344ccae2caf52c68ccb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260810 07:28:51.914654 17197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:51.914687 17197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:51.914736 17197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:51.915441 17197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569f6746234344ccae2caf52c68ccb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260810 07:28:51.915537 17197 leader_election.cc:302] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569f6746234344ccae2caf52c68ccb8e; no voters:
I20260810 07:28:51.915673 17197 leader_election.cc:288] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:51.915738 17207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:51.915915 17207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [term 1 LEADER]: Becoming Leader. State: Replica: 569f6746234344ccae2caf52c68ccb8e, State: Running, Role: LEADER
I20260810 07:28:51.915958 17197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:51.916008 17207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569f6746234344ccae2caf52c68ccb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260810 07:28:51.916410 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569f6746234344ccae2caf52c68ccb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569f6746234344ccae2caf52c68ccb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } } }
I20260810 07:28:51.916540 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:51.916409 17211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569f6746234344ccae2caf52c68ccb8e. Latest consensus state: current_term: 1 leader_uuid: "569f6746234344ccae2caf52c68ccb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569f6746234344ccae2caf52c68ccb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } } }
I20260810 07:28:51.916709 17211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:51.916821 17218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:51.917413 17218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:51.918797 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:51.914970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:50897 (local address 127.0.0.1:34421)
0810 07:28:51.915008 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:51.915010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:51.917573 (+ 2563us) server_negotiation.cc:407] Connection header received
0810 07:28:51.917725 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:51.917726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:51.917732 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:51.917794 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:51.918541 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:51.918542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:51.918543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:51.918547 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:51.918562 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:51.918566 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:51.918613 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:51.918718 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 07:28:51.918732 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 07:28:51.919304 17218 catalog_manager.cc:1347] Generated new cluster ID: 85e17b00ff954685b5dac1802ed382d5
I20260810 07:28:51.919377 17218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:51.932868 17218 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:51.933130 17218 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:51.952584 17218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569f6746234344ccae2caf52c68ccb8e: Generated new TSK 0
I20260810 07:28:51.952818 17218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:51.959971 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17119
2026-08-10T07:28:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:52 dist-test-slave-vxj2 krb5kdc[17231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:52 dist-test-slave-vxj2 krb5kdc[17231](info): set up 2 sockets
Aug 10 07:28:52 dist-test-slave-vxj2 krb5kdc[17231](info): commencing operation
krb5kdc: starting...
W20260810 07:28:53.084340 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:53 dist-test-slave-vxj2 krb5kdc[17231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:53.214113 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:53.299018 17245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:53.299243 17245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:53.299268 17245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:53.300518 17245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:53.300568 17245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:53.300583 17245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:53.300595 17245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:53.300608 17245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:53.302243 17245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:53.302536 17245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:53.302796 17245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:53.308038 17257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:53.307955 17254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:53.308025 17255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:53.308354 17245 server_base.cc:1034] running on GCE node
I20260810 07:28:53.308648 17245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:53.308990 17245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:53.310163 17245 hybrid_clock.cc:648] HybridClock initialized: now 1786346933310166 us; error 70 us; skew 500 ppm
Aug 10 07:28:53 dist-test-slave-vxj2 krb5kdc[17231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:53.312970 17245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:53.313499 17245 webserver.cc:460] Webserver started at http://127.0.0.1:34133/ using document root <none> and password file <none>
I20260810 07:28:53.313745 17245 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:53.313817 17245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:53.313966 17245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:53.315070 17245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "4834ed545a974ffd92a0e9550d304204"
format_stamp: "Formatted at 2026-08-10 07:28:53 on dist-test-slave-vxj2"
server_key: "019850d83ef844fb028f8dbea9fa9a71"
server_key_iv: "7f9d59e9898b8b4175b0481306cc597e"
server_key_version: "encryptionkey@0"
I20260810 07:28:53.315449 17245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "4834ed545a974ffd92a0e9550d304204"
format_stamp: "Formatted at 2026-08-10 07:28:53 on dist-test-slave-vxj2"
server_key: "019850d83ef844fb028f8dbea9fa9a71"
server_key_iv: "7f9d59e9898b8b4175b0481306cc597e"
server_key_version: "encryptionkey@0"
I20260810 07:28:53.316910 17245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:53.317781 17264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:53.317956 17245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:28:53.318063 17245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "4834ed545a974ffd92a0e9550d304204"
format_stamp: "Formatted at 2026-08-10 07:28:53 on dist-test-slave-vxj2"
server_key: "019850d83ef844fb028f8dbea9fa9a71"
server_key_iv: "7f9d59e9898b8b4175b0481306cc597e"
server_key_version: "encryptionkey@0"
I20260810 07:28:53.318147 17245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:53.330744 17245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:53.332747 17245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:53.332934 17245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:53.336771 17245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45861
I20260810 07:28:53.336814 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45861 every 8 connection(s)
I20260810 07:28:53.337275 17245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:53.338225 17321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:53.339640 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17245
I20260810 07:28:53.339781 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:53.339985 24819 external_mini_cluster.cc:1428] Setting key 2bb27af214d26ed128a5a79483d0b05b
I20260810 07:28:53.341041 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204: Bootstrap starting.
I20260810 07:28:53.341670 17321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:53.341979 17321 log.cc:824] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:53.342746 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204: No bootstrap required, opened a new log
I20260810 07:28:53.344954 17321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4834ed545a974ffd92a0e9550d304204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260810 07:28:53.345259 17321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:53.345325 17321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4834ed545a974ffd92a0e9550d304204, State: Initialized, Role: FOLLOWER
I20260810 07:28:53.345468 17321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4834ed545a974ffd92a0e9550d304204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260810 07:28:53.345559 17321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:53.345603 17321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:53.345659 17321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:53.346284 17321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4834ed545a974ffd92a0e9550d304204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260810 07:28:53.346374 17321 leader_election.cc:302] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4834ed545a974ffd92a0e9550d304204; no voters:
I20260810 07:28:53.346580 17321 leader_election.cc:288] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:53.346639 17326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:53.346819 17326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [term 1 LEADER]: Becoming Leader. State: Replica: 4834ed545a974ffd92a0e9550d304204, State: Running, Role: LEADER
I20260810 07:28:53.346871 17321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:53.346935 17326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4834ed545a974ffd92a0e9550d304204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260810 07:28:53.347272 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4834ed545a974ffd92a0e9550d304204. Latest consensus state: current_term: 1 leader_uuid: "4834ed545a974ffd92a0e9550d304204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4834ed545a974ffd92a0e9550d304204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } } }
I20260810 07:28:53.347283 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4834ed545a974ffd92a0e9550d304204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4834ed545a974ffd92a0e9550d304204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } } }
I20260810 07:28:53.347355 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:53.347400 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204 [sys.catalog]: This master's current role is: LEADER
Aug 10 07:28:53 dist-test-slave-vxj2 krb5kdc[17231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:28:53.348520 17345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:53.349210 17345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:53.350689 17345 catalog_manager.cc:1347] Generated new cluster ID: bbc435ed410940a5939cb7662c185c03
I20260810 07:28:53.350744 17345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:53.352083 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:53.340622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:59527 (local address 127.0.0.1:45861)
0810 07:28:53.340818 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:53.340827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:53.340854 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:28:53.341397 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:53.341405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:53.341459 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:53.341622 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:53.342363 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:53.343431 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:53.344267 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:53.344935 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:53.345067 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:53.348035 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:53.348039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:53.348040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:53.348053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:53.350256 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:53.350771 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:53.350774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:53.350775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:53.350881 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:53.351155 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:53.351156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:53.351157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:53.351481 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:53.351652 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:53.351766 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 07:28:53.351885 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260810 07:28:53.354236 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:53.363802 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:53.355798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:56707 (local address 127.0.0.1:45861)
0810 07:28:53.355873 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:53.355876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:53.356099 (+ 223us) server_negotiation.cc:407] Connection header received
0810 07:28:53.356161 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:53.356162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:53.356173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:53.356221 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:53.356929 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:53.357766 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:53.358870 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:53.359204 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:53.359263 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:53.360832 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:53.360833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:53.360834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:53.360841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:53.362499 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:53.362949 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:53.362950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:53.362951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:53.363009 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:53.363243 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:53.363244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:53.363244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:53.363369 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:53.363467 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:53.363644 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 07:28:53.363708 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:28:53.372931 17345 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:53.373751 17345 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:53.380645 17345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4834ed545a974ffd92a0e9550d304204: Generated new TSK 0
I20260810 07:28:53.381009 17345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:53.383745 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17245
W20260810 07:28:53.392956 17229 connection.cc:535] client connection to 127.0.0.1:45861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45861): Connection reset by peer (error 104)
2026-08-10T07:28:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:53 dist-test-slave-vxj2 krb5kdc[17358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:53 dist-test-slave-vxj2 krb5kdc[17358](info): set up 2 sockets
Aug 10 07:28:53 dist-test-slave-vxj2 krb5kdc[17358](info): commencing operation
krb5kdc: starting...
W20260810 07:28:54.539172 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:54 dist-test-slave-vxj2 krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:54.673266 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:54.759469 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:54.759722 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:54.759750 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:54.760979 17374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:54.761032 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:54.761049 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:54.761065 17374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:54.761126 17374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:54.762765 17374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:54.763064 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:54.763331 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:54.768699 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:54.768750 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:54.768738 17382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:54.769016 17374 server_base.cc:1034] running on GCE node
I20260810 07:28:54.769338 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:54.769618 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:54.770816 17374 hybrid_clock.cc:648] HybridClock initialized: now 1786346934770735 us; error 54 us; skew 500 ppm
Aug 10 07:28:54 dist-test-slave-vxj2 krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:54.773644 17374 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:54.774250 17374 webserver.cc:460] Webserver started at http://127.0.0.1:45613/ using document root <none> and password file <none>
I20260810 07:28:54.774547 17374 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:54.774631 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:54.774812 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:54.776014 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "33f0055469e14e53a5444913e1ca8c84"
format_stamp: "Formatted at 2026-08-10 07:28:54 on dist-test-slave-vxj2"
server_key: "8f8e5bd3679021f2b934bef275f11d50"
server_key_iv: "09d232d8990f00835bbb8645a6c126d7"
server_key_version: "encryptionkey@0"
I20260810 07:28:54.776401 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "33f0055469e14e53a5444913e1ca8c84"
format_stamp: "Formatted at 2026-08-10 07:28:54 on dist-test-slave-vxj2"
server_key: "8f8e5bd3679021f2b934bef275f11d50"
server_key_iv: "09d232d8990f00835bbb8645a6c126d7"
server_key_version: "encryptionkey@0"
I20260810 07:28:54.777951 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:28:54.778774 17392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:54.778988 17374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:54.779109 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "33f0055469e14e53a5444913e1ca8c84"
format_stamp: "Formatted at 2026-08-10 07:28:54 on dist-test-slave-vxj2"
server_key: "8f8e5bd3679021f2b934bef275f11d50"
server_key_iv: "09d232d8990f00835bbb8645a6c126d7"
server_key_version: "encryptionkey@0"
I20260810 07:28:54.779191 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:54.794755 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:54.797369 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:54.797562 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:54.801930 17374 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40141
I20260810 07:28:54.802016 17450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40141 every 8 connection(s)
I20260810 07:28:54.802392 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:54.803319 17451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:54.805755 17451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84: Bootstrap starting.
I20260810 07:28:54.806408 17451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:54.806723 17451 log.cc:824] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:54.807497 17451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84: No bootstrap required, opened a new log
I20260810 07:28:54.809276 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17374
I20260810 07:28:54.809278 17451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f0055469e14e53a5444913e1ca8c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260810 07:28:54.809450 17451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:54.809473 17451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f0055469e14e53a5444913e1ca8c84, State: Initialized, Role: FOLLOWER
I20260810 07:28:54.809515 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:54.809664 17451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f0055469e14e53a5444913e1ca8c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260810 07:28:54.809731 17451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:54.809772 17451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:54.809813 24819 external_mini_cluster.cc:1428] Setting key a5a471f94dba0bd8931e94d85fdb377a
I20260810 07:28:54.809825 17451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:54.810462 17451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f0055469e14e53a5444913e1ca8c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260810 07:28:54.810617 17451 leader_election.cc:302] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33f0055469e14e53a5444913e1ca8c84; no voters:
I20260810 07:28:54.810813 17451 leader_election.cc:288] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:54.810854 17457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:54.811008 17457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [term 1 LEADER]: Becoming Leader. State: Replica: 33f0055469e14e53a5444913e1ca8c84, State: Running, Role: LEADER
I20260810 07:28:54.811110 17451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:54.811124 17457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f0055469e14e53a5444913e1ca8c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } }
I20260810 07:28:54.811497 17457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33f0055469e14e53a5444913e1ca8c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f0055469e14e53a5444913e1ca8c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } } }
I20260810 07:28:54.811648 17457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:54.811834 17457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33f0055469e14e53a5444913e1ca8c84. Latest consensus state: current_term: 1 leader_uuid: "33f0055469e14e53a5444913e1ca8c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f0055469e14e53a5444913e1ca8c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40141 } } }
I20260810 07:28:54.811954 17457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:54.812338 17466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:54.813068 17466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:54.814893 17466 catalog_manager.cc:1347] Generated new cluster ID: 27a67ede680c4371835c315929013788
I20260810 07:28:54.815313 17466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:54.816215 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.810380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:34659 (local address 127.0.0.1:40141)
0810 07:28:54.810594 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:54.810605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:54.810639 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:28:54.810788 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:54.810796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:54.810884 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:54.811031 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:54.811322 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.813029 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.813936 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.814551 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.814674 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:54.815469 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:54.815473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:54.815473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:54.815676 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:54.815712 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:54.815726 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:54.815809 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:54.815985 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 07:28:54.816015 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260810 07:28:54.817273 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:54.829061 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.821599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:43783 (local address 127.0.0.1:40141)
0810 07:28:54.821636 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:54.821638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:54.825385 (+ 3747us) server_negotiation.cc:407] Connection header received
0810 07:28:54.825461 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:54.825462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:54.825475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:54.825542 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:54.825725 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.826485 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.827588 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.827934 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.827995 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:54.828764 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:54.828765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:54.828766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:54.828771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:54.828794 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:54.828799 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:54.828853 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:54.828958 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 07:28:54.828972 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 07:28:54.840646 17466 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:54.841452 17466 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:54.845698 17466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33f0055469e14e53a5444913e1ca8c84: Generated new TSK 0
I20260810 07:28:54.846134 17466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:54.856191 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17374
W20260810 07:28:54.864917 17356 connection.cc:535] client connection to 127.0.0.1:40141 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40141): Connection reset by peer (error 104)
2026-08-10T07:28:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:54 dist-test-slave-vxj2 krb5kdc[17486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:54 dist-test-slave-vxj2 krb5kdc[17486](info): set up 2 sockets
Aug 10 07:28:54 dist-test-slave-vxj2 krb5kdc[17486](info): commencing operation
krb5kdc: starting...
W20260810 07:28:55.978343 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:56 dist-test-slave-vxj2 krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:28:56.104259 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:56.189445 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:56.189709 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:56.189746 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:56.190922 17501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:56.191007 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:56.191036 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:56.191067 17501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:56.191092 17501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:56.192624 17501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:56.192915 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:56.193186 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:56.198652 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:56.198624 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:56.198635 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:56.198675 17501 server_base.cc:1034] running on GCE node
I20260810 07:28:56.199193 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:56.199532 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:56.200678 17501 hybrid_clock.cc:648] HybridClock initialized: now 1786346936200665 us; error 31 us; skew 500 ppm
Aug 10 07:28:56 dist-test-slave-vxj2 krb5kdc[17486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:56.203311 17501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:28:56.203807 17501 webserver.cc:460] Webserver started at http://127.0.0.1:44789/ using document root <none> and password file <none>
I20260810 07:28:56.204082 17501 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:56.204154 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:56.204304 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:56.205510 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "8f52ce3628c44aab8289cb33473bea91"
format_stamp: "Formatted at 2026-08-10 07:28:56 on dist-test-slave-vxj2"
server_key: "5e21b1bf0c0c971412b1b077696f5724"
server_key_iv: "e2a8193ccb7fa942765399037d38ebdd"
server_key_version: "encryptionkey@0"
I20260810 07:28:56.205878 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "8f52ce3628c44aab8289cb33473bea91"
format_stamp: "Formatted at 2026-08-10 07:28:56 on dist-test-slave-vxj2"
server_key: "5e21b1bf0c0c971412b1b077696f5724"
server_key_iv: "e2a8193ccb7fa942765399037d38ebdd"
server_key_version: "encryptionkey@0"
I20260810 07:28:56.207333 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 07:28:56.208148 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:56.208343 17501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:56.208473 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "8f52ce3628c44aab8289cb33473bea91"
format_stamp: "Formatted at 2026-08-10 07:28:56 on dist-test-slave-vxj2"
server_key: "5e21b1bf0c0c971412b1b077696f5724"
server_key_iv: "e2a8193ccb7fa942765399037d38ebdd"
server_key_version: "encryptionkey@0"
I20260810 07:28:56.208554 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:56.212059 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:56.214430 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:56.214601 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:56.218595 17501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42319
I20260810 07:28:56.218631 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42319 every 8 connection(s)
I20260810 07:28:56.218974 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:56.219628 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17501
I20260810 07:28:56.219786 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:56.220016 24819 external_mini_cluster.cc:1428] Setting key 740b9b952626bd3e389b9a5d43457d0e
I20260810 07:28:56.220275 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:56.222987 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:56.220612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:51287 (local address 127.0.0.1:42319)
0810 07:28:56.220843 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:56.220852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:56.220879 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:28:56.221486 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:56.221494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:56.221522 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:56.221678 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:56.222422 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:56.222425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:56.222426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:56.222432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:56.222475 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:56.222487 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:56.222565 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:56.222636 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 07:28:56.222832 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260810 07:28:56.223704 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91: Bootstrap starting.
I20260810 07:28:56.224418 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:56.224498 17578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:56.224802 17578 log.cc:824] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:56.225747 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91: No bootstrap required, opened a new log
I20260810 07:28:56.227424 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:56.226237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:33965 (local address 127.0.0.1:42319)
0810 07:28:56.226280 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:56.226282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:56.226383 (+ 101us) server_negotiation.cc:407] Connection header received
0810 07:28:56.226470 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:56.226470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:56.226476 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:56.226557 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:56.227183 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:56.227185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:56.227185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:56.227189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:56.227204 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:56.227208 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:56.227278 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:56.227337 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 07:28:56.227352 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:28:56.228188 17578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f52ce3628c44aab8289cb33473bea91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42319 } }
I20260810 07:28:56.228330 17578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:56.228355 17578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f52ce3628c44aab8289cb33473bea91, State: Initialized, Role: FOLLOWER
I20260810 07:28:56.228504 17578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f52ce3628c44aab8289cb33473bea91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42319 } }
I20260810 07:28:56.228582 17578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:56.228627 17578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:56.228680 17578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:56.229395 17578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f52ce3628c44aab8289cb33473bea91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42319 } }
I20260810 07:28:56.229487 17578 leader_election.cc:302] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f52ce3628c44aab8289cb33473bea91; no voters:
I20260810 07:28:56.229661 17578 leader_election.cc:288] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:56.229763 17589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:56.229959 17578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:56.230018 17589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [term 1 LEADER]: Becoming Leader. State: Replica: 8f52ce3628c44aab8289cb33473bea91, State: Running, Role: LEADER
I20260810 07:28:56.230295 17589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f52ce3628c44aab8289cb33473bea91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42319 } }
I20260810 07:28:56.230756 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f52ce3628c44aab8289cb33473bea91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f52ce3628c44aab8289cb33473bea91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42319 } } }
I20260810 07:28:56.230752 17591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f52ce3628c44aab8289cb33473bea91. Latest consensus state: current_term: 1 leader_uuid: "8f52ce3628c44aab8289cb33473bea91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f52ce3628c44aab8289cb33473bea91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42319 } } }
I20260810 07:28:56.230895 17591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:56.231066 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:56.231452 17599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:56.231967 17599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:56.233445 17599 catalog_manager.cc:1347] Generated new cluster ID: b6d2f4e848a94bdd98303304eb0b9823
I20260810 07:28:56.233501 17599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:56.240154 17599 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:56.240352 17599 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:56.257532 17599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f52ce3628c44aab8289cb33473bea91: Generated new TSK 0
I20260810 07:28:56.257797 17599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:56.267242 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17501
2026-08-10T07:28:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:56 dist-test-slave-vxj2 krb5kdc[17610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:56 dist-test-slave-vxj2 krb5kdc[17610](info): set up 2 sockets
Aug 10 07:28:56 dist-test-slave-vxj2 krb5kdc[17610](info): commencing operation
krb5kdc: starting...
W20260810 07:28:57.388955 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:57 dist-test-slave-vxj2 krb5kdc[17610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:28:57.516495 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35527
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:57.600543 17623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:57.600819 17623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:57.600862 17623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:57.602077 17623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:57.602159 17623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:57.602185 17623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:57.602206 17623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:57.602240 17623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:57.603742 17623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:35527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:57.604049 17623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:57.604290 17623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:57.609881 17629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:57.609949 17628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:57.610016 17631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:57.610275 17623 server_base.cc:1034] running on GCE node
I20260810 07:28:57.610545 17623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:57.610844 17623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:57.612039 17623 hybrid_clock.cc:648] HybridClock initialized: now 1786346937612011 us; error 44 us; skew 500 ppm
Aug 10 07:28:57 dist-test-slave-vxj2 krb5kdc[17610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346937, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:57.619715 17623 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:28:57.620314 17623 webserver.cc:460] Webserver started at http://127.24.60.254:44187/ using document root <none> and password file <none>
I20260810 07:28:57.620573 17623 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:57.620645 17623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:57.620800 17623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:57.622004 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "9256ee034ac54eefa607e65286b90fd6"
format_stamp: "Formatted at 2026-08-10 07:28:57 on dist-test-slave-vxj2"
server_key: "1770d9445a1a8c40ba47c61e5fbe9b84"
server_key_iv: "f84ecc25623d2371d3cb0ba37dfa8dd6"
server_key_version: "encryptionkey@0"
I20260810 07:28:57.622392 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "9256ee034ac54eefa607e65286b90fd6"
format_stamp: "Formatted at 2026-08-10 07:28:57 on dist-test-slave-vxj2"
server_key: "1770d9445a1a8c40ba47c61e5fbe9b84"
server_key_iv: "f84ecc25623d2371d3cb0ba37dfa8dd6"
server_key_version: "encryptionkey@0"
I20260810 07:28:57.623798 17623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:28:57.624590 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:57.624774 17623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 07:28:57.624915 17623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "9256ee034ac54eefa607e65286b90fd6"
format_stamp: "Formatted at 2026-08-10 07:28:57 on dist-test-slave-vxj2"
server_key: "1770d9445a1a8c40ba47c61e5fbe9b84"
server_key_iv: "f84ecc25623d2371d3cb0ba37dfa8dd6"
server_key_version: "encryptionkey@0"
I20260810 07:28:57.625001 17623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:57.637946 17623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:57.640288 17623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:57.640468 17623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:57.644804 17623 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:35527
I20260810 07:28:57.644846 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:35527 every 8 connection(s)
I20260810 07:28:57.645309 17623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:57.646260 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:57.648587 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6: Bootstrap starting.
I20260810 07:28:57.649123 17701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:57.649471 17701 log.cc:824] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:57.650250 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6: No bootstrap required, opened a new log
I20260810 07:28:57.651896 17701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ee034ac54eefa607e65286b90fd6" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35527 } }
I20260810 07:28:57.652098 17701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:57.652127 17701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9256ee034ac54eefa607e65286b90fd6, State: Initialized, Role: FOLLOWER
I20260810 07:28:57.652297 17701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ee034ac54eefa607e65286b90fd6" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35527 } }
I20260810 07:28:57.652396 17701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:57.652421 17701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:57.652392 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17623
I20260810 07:28:57.652480 17701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:57.652534 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:57.652724 24819 external_mini_cluster.cc:1428] Setting key 3d5af36e7030a66a906dec347594b1ae
I20260810 07:28:57.653678 17701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ee034ac54eefa607e65286b90fd6" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35527 } }
I20260810 07:28:57.653816 17701 leader_election.cc:302] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9256ee034ac54eefa607e65286b90fd6; no voters:
I20260810 07:28:57.654027 17701 leader_election.cc:288] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:57.654129 17707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:57.654302 17707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 9256ee034ac54eefa607e65286b90fd6, State: Running, Role: LEADER
I20260810 07:28:57.654325 17701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:57.654403 17707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ee034ac54eefa607e65286b90fd6" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35527 } }
I20260810 07:28:57.654770 17707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9256ee034ac54eefa607e65286b90fd6. Latest consensus state: current_term: 1 leader_uuid: "9256ee034ac54eefa607e65286b90fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ee034ac54eefa607e65286b90fd6" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35527 } } }
I20260810 07:28:57.654776 17709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9256ee034ac54eefa607e65286b90fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ee034ac54eefa607e65286b90fd6" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35527 } } }
I20260810 07:28:57.654886 17707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:57.654939 17709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:57.655784 17717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:57.656378 17717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:57.658458 17717 catalog_manager.cc:1347] Generated new cluster ID: 73e7d1e99b4f4e8f8b921dfb81fd0d88
I20260810 07:28:57.658531 17717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:28:57 dist-test-slave-vxj2 krb5kdc[17610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:28:57.665656 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:57.653356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55737 (local address 127.24.60.254:35527)
0810 07:28:57.653566 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:57.653574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:57.653628 (+ 54us) server_negotiation.cc:407] Connection header received
0810 07:28:57.653809 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:57.653817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:57.653899 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:57.654078 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:57.655136 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:57.656655 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:57.657745 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:57.658246 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:57.658332 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:57.661358 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:57.661362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:57.661363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:57.661376 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:57.663619 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:57.664094 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:57.664098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:57.664098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:57.664259 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:57.664525 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:57.664526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:57.664526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:57.664979 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:57.665186 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:57.665331 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 07:28:57.665444 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260810 07:28:57.667070 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:57.676056 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:57.668564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34201 (local address 127.24.60.254:35527)
0810 07:28:57.668612 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:57.668614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:57.668651 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:28:57.668729 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:57.668730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:57.668740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:57.668789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:57.669508 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:57.670241 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:57.671264 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:57.671606 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:57.671666 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:57.673148 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:57.673169 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:57.673170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:57.673177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:57.674766 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:57.675210 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:57.675211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:57.675211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:57.675268 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:57.675506 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:57.675507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:57.675507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:57.675623 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:57.675735 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:57.675900 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 07:28:57.675964 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:28:57.678251 17717 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:57.678737 17717 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:57.686463 17717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9256ee034ac54eefa607e65286b90fd6: Generated new TSK 0
I20260810 07:28:57.686709 17717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:57.693392 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17623
W20260810 07:28:57.701779 17608 connection.cc:535] client connection to 127.24.60.254:35527 recv error: Network error: failed to read from TLS socket (remote: 127.24.60.254:35527): Connection reset by peer (error 104)
2026-08-10T07:28:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:57 dist-test-slave-vxj2 krb5kdc[17739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:57 dist-test-slave-vxj2 krb5kdc[17739](info): set up 2 sockets
Aug 10 07:28:57 dist-test-slave-vxj2 krb5kdc[17739](info): commencing operation
krb5kdc: starting...
W20260810 07:28:58.811091 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:28:58 dist-test-slave-vxj2 krb5kdc[17739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:28:58.937338 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35237
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:35237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:59.021523 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:59.021814 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:59.021859 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:59.023144 17753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:59.023231 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:59.023255 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:59.023288 17753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:59.023315 17753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:59.024888 17753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:35237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:35237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:28:59.025238 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:59.025487 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:59.030874 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:59.030895 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:59.030874 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:59.031201 17753 server_base.cc:1034] running on GCE node
I20260810 07:28:59.031522 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:59.031785 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:59.033004 17753 hybrid_clock.cc:648] HybridClock initialized: now 1786346939033015 us; error 57 us; skew 500 ppm
Aug 10 07:28:59 dist-test-slave-vxj2 krb5kdc[17739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346939, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:59.035779 17753 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:28:59.036300 17753 webserver.cc:460] Webserver started at http://127.24.60.254:36493/ using document root <none> and password file <none>
I20260810 07:28:59.036588 17753 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:59.036661 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:59.036865 17753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:59.038067 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "5127b2c18d894dbe8c80c0cdb607793a"
format_stamp: "Formatted at 2026-08-10 07:28:59 on dist-test-slave-vxj2"
server_key: "2b28d0e93bb9ef5e0a59746d21e5e006"
server_key_iv: "fa5eb585367b73dc21a8185965314f5a"
server_key_version: "encryptionkey@0"
I20260810 07:28:59.038455 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "5127b2c18d894dbe8c80c0cdb607793a"
format_stamp: "Formatted at 2026-08-10 07:28:59 on dist-test-slave-vxj2"
server_key: "2b28d0e93bb9ef5e0a59746d21e5e006"
server_key_iv: "fa5eb585367b73dc21a8185965314f5a"
server_key_version: "encryptionkey@0"
I20260810 07:28:59.039898 17753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 07:28:59.040750 17770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:59.040968 17753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:59.041098 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "5127b2c18d894dbe8c80c0cdb607793a"
format_stamp: "Formatted at 2026-08-10 07:28:59 on dist-test-slave-vxj2"
server_key: "2b28d0e93bb9ef5e0a59746d21e5e006"
server_key_iv: "fa5eb585367b73dc21a8185965314f5a"
server_key_version: "encryptionkey@0"
I20260810 07:28:59.041216 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:59.064944 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:59.067320 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:59.067531 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:59.071911 17753 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:35237
I20260810 07:28:59.071951 17831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:35237 every 8 connection(s)
I20260810 07:28:59.072319 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:59.072665 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17753
I20260810 07:28:59.072806 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:28:59.073016 24819 external_mini_cluster.cc:1428] Setting key 0102fac31193c57420735e470bcfca2c
I20260810 07:28:59.073411 17832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:59.076973 17832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a: Bootstrap starting.
I20260810 07:28:59.077737 17832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:59.078107 17832 log.cc:824] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:59.078986 17832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a: No bootstrap required, opened a new log
I20260810 07:28:59.079329 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:59.073620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57123 (local address 127.24.60.254:35237)
0810 07:28:59.073819 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:59.073828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:59.073854 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:28:59.074544 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:59.074551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:59.074600 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:59.074724 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:59.075090 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:59.075993 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:59.077235 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:59.077646 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:59.077733 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:59.078580 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:59.078585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:59.078586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:59.078798 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:59.078830 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:59.078842 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:59.078917 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:59.079091 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 07:28:59.079125 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260810 07:28:59.080581 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:28:59.081377 17832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5127b2c18d894dbe8c80c0cdb607793a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35237 } }
I20260810 07:28:59.081518 17832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:59.081562 17832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5127b2c18d894dbe8c80c0cdb607793a, State: Initialized, Role: FOLLOWER
I20260810 07:28:59.081736 17832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5127b2c18d894dbe8c80c0cdb607793a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35237 } }
I20260810 07:28:59.081818 17832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:59.081841 17832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:59.081902 17832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:59.082700 17832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5127b2c18d894dbe8c80c0cdb607793a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35237 } }
I20260810 07:28:59.082791 17832 leader_election.cc:302] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5127b2c18d894dbe8c80c0cdb607793a; no voters:
I20260810 07:28:59.082975 17832 leader_election.cc:288] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:59.083001 17845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:59.083114 17845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [term 1 LEADER]: Becoming Leader. State: Replica: 5127b2c18d894dbe8c80c0cdb607793a, State: Running, Role: LEADER
I20260810 07:28:59.083243 17832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:59.083252 17845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5127b2c18d894dbe8c80c0cdb607793a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35237 } }
I20260810 07:28:59.083772 17846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5127b2c18d894dbe8c80c0cdb607793a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5127b2c18d894dbe8c80c0cdb607793a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35237 } } }
I20260810 07:28:59.083988 17846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:59.083781 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5127b2c18d894dbe8c80c0cdb607793a. Latest consensus state: current_term: 1 leader_uuid: "5127b2c18d894dbe8c80c0cdb607793a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5127b2c18d894dbe8c80c0cdb607793a" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 35237 } } }
I20260810 07:28:59.084226 17854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:59.084254 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:59.084744 17854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:59.085755 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:59.082123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35017 (local address 127.24.60.254:35237)
0810 07:28:59.082167 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:59.082169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:59.082260 (+ 91us) server_negotiation.cc:407] Connection header received
0810 07:28:59.082340 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:59.082341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:59.082349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:59.082389 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:59.082549 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:59.083171 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:59.084208 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:59.084510 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:59.084880 (+ 370us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:59.085378 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:59.085379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:59.085380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:28:59.085384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:59.085400 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:28:59.085404 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:59.085459 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:59.085679 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 07:28:59.085689 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 07:28:59.086517 17854 catalog_manager.cc:1347] Generated new cluster ID: 49300124134a4494af551f28b5c39f47
I20260810 07:28:59.086570 17854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:59.091758 17854 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:59.092470 17854 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:59.095525 17854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5127b2c18d894dbe8c80c0cdb607793a: Generated new TSK 0
I20260810 07:28:59.095693 17854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:28:59.101039 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17753
W20260810 07:28:59.108641 17737 connection.cc:535] client connection to 127.24.60.254:35237 recv error: Network error: failed to read from TLS socket (remote: 127.24.60.254:35237): Connection reset by peer (error 104)
2026-08-10T07:28:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:59 dist-test-slave-vxj2 krb5kdc[17867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:59 dist-test-slave-vxj2 krb5kdc[17867](info): set up 2 sockets
Aug 10 07:28:59 dist-test-slave-vxj2 krb5kdc[17867](info): commencing operation
krb5kdc: starting...
W20260810 07:29:00.278095 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:29:00 dist-test-slave-vxj2 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:29:00.410395 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:46615
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:00.495139 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:00.495404 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:00.495445 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:00.496613 17882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:00.496688 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:00.496711 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:00.496731 17882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:00.496762 17882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:00.498360 17882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:46615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:00.498646 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:00.498888 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:00.504086 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:00.504086 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:00.504117 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:00.504451 17882 server_base.cc:1034] running on GCE node
I20260810 07:29:00.504720 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:00.505067 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:00.506270 17882 hybrid_clock.cc:648] HybridClock initialized: now 1786346940506256 us; error 28 us; skew 500 ppm
Aug 10 07:29:00 dist-test-slave-vxj2 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346940, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:00.508945 17882 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:29:00.509514 17882 webserver.cc:460] Webserver started at http://127.24.60.254:46195/ using document root <none> and password file <none>
I20260810 07:29:00.509775 17882 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:00.509847 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:00.510006 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:00.511107 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "d3dce478798b43b981f593613617a6d5"
format_stamp: "Formatted at 2026-08-10 07:29:00 on dist-test-slave-vxj2"
server_key: "cb07b98ff19256dcd1a11f2cb791403d"
server_key_iv: "3e8eebe9d9cb4b62fe9b666f4e32a9a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:00.511479 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "d3dce478798b43b981f593613617a6d5"
format_stamp: "Formatted at 2026-08-10 07:29:00 on dist-test-slave-vxj2"
server_key: "cb07b98ff19256dcd1a11f2cb791403d"
server_key_iv: "3e8eebe9d9cb4b62fe9b666f4e32a9a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:00.512885 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:29:00.513765 17897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:00.513978 17882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:29:00.514089 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "d3dce478798b43b981f593613617a6d5"
format_stamp: "Formatted at 2026-08-10 07:29:00 on dist-test-slave-vxj2"
server_key: "cb07b98ff19256dcd1a11f2cb791403d"
server_key_iv: "3e8eebe9d9cb4b62fe9b666f4e32a9a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:00.514176 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:00.517661 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:00.519608 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:00.519778 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:00.523816 17882 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:46615
I20260810 07:29:00.523912 17955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:46615 every 8 connection(s)
I20260810 07:29:00.524230 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:00.525406 17956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:00.526117 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 17882
I20260810 07:29:00.526240 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:29:00.526435 24819 external_mini_cluster.cc:1428] Setting key e12d93a5dbb87cf6fb8b35069dbb6a17
I20260810 07:29:00.528502 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5: Bootstrap starting.
I20260810 07:29:00.529006 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:00.526989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44321 (local address 127.24.60.254:46615)
0810 07:29:00.527198 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:00.527206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:00.527232 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:29:00.527652 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:00.527659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:00.527684 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:00.527836 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:00.528572 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:00.528576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:00.528576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:00.528580 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:00.528616 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:00.528626 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:00.528697 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:00.528777 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 07:29:00.528820 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260810 07:29:00.529321 17956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:00.529662 17956 log.cc:824] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:00.529968 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:00.530413 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5: No bootstrap required, opened a new log
I20260810 07:29:00.532588 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:00.531455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40633 (local address 127.24.60.254:46615)
0810 07:29:00.531509 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:00.531512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:00.531583 (+ 71us) server_negotiation.cc:407] Connection header received
0810 07:29:00.531666 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:00.531667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:00.531671 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:00.531742 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:00.532344 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:00.532345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:00.532345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:00.532350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:00.532362 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:00.532366 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:00.532421 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:00.532502 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 07:29:00.532517 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:29:00.532827 17956 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dce478798b43b981f593613617a6d5" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 46615 } }
I20260810 07:29:00.532971 17956 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:00.533023 17956 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dce478798b43b981f593613617a6d5, State: Initialized, Role: FOLLOWER
I20260810 07:29:00.533146 17956 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dce478798b43b981f593613617a6d5" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 46615 } }
I20260810 07:29:00.533288 17956 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:00.533339 17956 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:00.533391 17956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:00.533941 17956 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dce478798b43b981f593613617a6d5" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 46615 } }
I20260810 07:29:00.534058 17956 leader_election.cc:302] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3dce478798b43b981f593613617a6d5; no voters:
I20260810 07:29:00.534341 17956 leader_election.cc:288] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:00.534583 17968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:00.534760 17968 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [term 1 LEADER]: Becoming Leader. State: Replica: d3dce478798b43b981f593613617a6d5, State: Running, Role: LEADER
I20260810 07:29:00.534924 17968 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dce478798b43b981f593613617a6d5" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 46615 } }
I20260810 07:29:00.534760 17956 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:00.535401 17970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3dce478798b43b981f593613617a6d5. Latest consensus state: current_term: 1 leader_uuid: "d3dce478798b43b981f593613617a6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dce478798b43b981f593613617a6d5" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 46615 } } }
I20260810 07:29:00.535455 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3dce478798b43b981f593613617a6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dce478798b43b981f593613617a6d5" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 46615 } } }
I20260810 07:29:00.535640 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:00.535554 17970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:00.535953 17972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:00.536563 17972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:00.538345 17972 catalog_manager.cc:1347] Generated new cluster ID: 935edfcf0883481a8fb07ebf1b455a02
I20260810 07:29:00.538429 17972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:00.549284 17972 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:00.549510 17972 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:00.552225 17972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3dce478798b43b981f593613617a6d5: Generated new TSK 0
I20260810 07:29:00.552404 17972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:00.558712 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 17882
2026-08-10T07:29:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:00 dist-test-slave-vxj2 krb5kdc[17993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:00 dist-test-slave-vxj2 krb5kdc[17993](info): set up 2 sockets
Aug 10 07:29:00 dist-test-slave-vxj2 krb5kdc[17993](info): commencing operation
krb5kdc: starting...
W20260810 07:29:01.680140 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:29:01 dist-test-slave-vxj2 krb5kdc[17993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:29:01.824932 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:37231
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:01.909430 18010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:01.909648 18010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:01.909672 18010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:01.910959 18010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:01.911007 18010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:01.911022 18010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:01.911036 18010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:01.911047 18010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:01.912582 18010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:01.912860 18010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:01.913107 18010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:01.918143 18016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:01.918133 18015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:01.918154 18010 server_base.cc:1034] running on GCE node
W20260810 07:29:01.918133 18019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:01.918624 18010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:01.918918 18010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:01.920061 18010 hybrid_clock.cc:648] HybridClock initialized: now 1786346941920053 us; error 35 us; skew 500 ppm
Aug 10 07:29:01 dist-test-slave-vxj2 krb5kdc[17993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346941, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:01.922709 18010 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:29:01.923182 18010 webserver.cc:460] Webserver started at http://127.24.60.254:40059/ using document root <none> and password file <none>
I20260810 07:29:01.923367 18010 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:01.923408 18010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:01.923503 18010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:01.924533 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "0c1449e1f3804f97b0df3c917b132430"
format_stamp: "Formatted at 2026-08-10 07:29:01 on dist-test-slave-vxj2"
server_key: "8a2509bdca0aa6f7f2734e4216d7827a"
server_key_iv: "ed2c04fcd8e07e5ee380f9c67fd321ac"
server_key_version: "encryptionkey@0"
I20260810 07:29:01.924835 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "0c1449e1f3804f97b0df3c917b132430"
format_stamp: "Formatted at 2026-08-10 07:29:01 on dist-test-slave-vxj2"
server_key: "8a2509bdca0aa6f7f2734e4216d7827a"
server_key_iv: "ed2c04fcd8e07e5ee380f9c67fd321ac"
server_key_version: "encryptionkey@0"
I20260810 07:29:01.926258 18010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:29:01.927028 18026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:01.927210 18010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:29:01.927321 18010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "0c1449e1f3804f97b0df3c917b132430"
format_stamp: "Formatted at 2026-08-10 07:29:01 on dist-test-slave-vxj2"
server_key: "8a2509bdca0aa6f7f2734e4216d7827a"
server_key_iv: "ed2c04fcd8e07e5ee380f9c67fd321ac"
server_key_version: "encryptionkey@0"
I20260810 07:29:01.927403 18010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:01.953752 18010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:01.956290 18010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:01.956480 18010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:01.960959 18010 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:37231
I20260810 07:29:01.960983 18082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:37231 every 8 connection(s)
I20260810 07:29:01.961408 18010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:01.962368 18083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:01.964735 18083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430: Bootstrap starting.
I20260810 07:29:01.965409 18083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:01.965770 18083 log.cc:824] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:01.966527 18083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430: No bootstrap required, opened a new log
I20260810 07:29:01.968142 18083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1449e1f3804f97b0df3c917b132430" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37231 } }
I20260810 07:29:01.968290 18083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:01.968314 18083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1449e1f3804f97b0df3c917b132430, State: Initialized, Role: FOLLOWER
I20260810 07:29:01.968497 18083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1449e1f3804f97b0df3c917b132430" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37231 } }
I20260810 07:29:01.968564 18083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:01.968623 18083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:01.968676 18083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:01.969662 18083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1449e1f3804f97b0df3c917b132430" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37231 } }
I20260810 07:29:01.969754 18083 leader_election.cc:302] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1449e1f3804f97b0df3c917b132430; no voters:
I20260810 07:29:01.969961 18083 leader_election.cc:288] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:01.970027 18086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:01.970207 18086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1449e1f3804f97b0df3c917b132430, State: Running, Role: LEADER
I20260810 07:29:01.970239 18083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:01.970345 18086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1449e1f3804f97b0df3c917b132430" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37231 } }
I20260810 07:29:01.970819 18087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c1449e1f3804f97b0df3c917b132430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1449e1f3804f97b0df3c917b132430" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37231 } } }
I20260810 07:29:01.971009 18087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:01.971083 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18010
I20260810 07:29:01.971206 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:29:01.971402 24819 external_mini_cluster.cc:1428] Setting key a00f2397e0208cddd85964683cfda850
I20260810 07:29:01.971474 18098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:01.970829 18088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1449e1f3804f97b0df3c917b132430. Latest consensus state: current_term: 1 leader_uuid: "0c1449e1f3804f97b0df3c917b132430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1449e1f3804f97b0df3c917b132430" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 37231 } } }
I20260810 07:29:01.971762 18088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:01.972021 18098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:01.973896 18098 catalog_manager.cc:1347] Generated new cluster ID: 27cda0124963482a8c7c040bb27162d6
I20260810 07:29:01.973950 18098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 07:29:01 dist-test-slave-vxj2 krb5kdc[17993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:01.984076 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:01.972181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59043 (local address 127.24.60.254:37231)
0810 07:29:01.972358 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:01.972368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:01.972404 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:29:01.972741 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:01.972750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:01.972830 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:01.973000 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:01.973859 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:01.975358 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:01.976501 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:01.977132 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:01.977257 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:01.980150 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:01.980154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:01.980155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:01.980167 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:01.982314 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:01.982739 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:01.982742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:01.982742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:01.982868 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:01.983095 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:01.983095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:01.983096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:01.983409 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:01.983560 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:01.983691 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 07:29:01.983833 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260810 07:29:01.985374 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:01.994161 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:01.986850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44423 (local address 127.24.60.254:37231)
0810 07:29:01.986884 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:01.986886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:01.987017 (+ 131us) server_negotiation.cc:407] Connection header received
0810 07:29:01.987106 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:01.987106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:01.987115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:01.987142 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:01.987801 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:01.988561 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:01.989586 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:01.989878 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:01.989915 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:01.991403 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:01.991404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:01.991404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:01.991411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:01.992946 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:01.993356 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:01.993357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:01.993357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:01.993411 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:01.993583 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:01.993584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:01.993584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:01.993697 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:01.993771 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:01.993984 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 07:29:01.994065 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 07:29:01.999478 18098 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:02.000265 18098 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:02.010933 18098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c1449e1f3804f97b0df3c917b132430: Generated new TSK 0
I20260810 07:29:02.011185 18098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:02.015625 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18010
2026-08-10T07:29:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:02 dist-test-slave-vxj2 krb5kdc[18120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:02 dist-test-slave-vxj2 krb5kdc[18120](info): set up 2 sockets
Aug 10 07:29:02 dist-test-slave-vxj2 krb5kdc[18120](info): commencing operation
krb5kdc: starting...
W20260810 07:29:03.147018 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:29:03 dist-test-slave-vxj2 krb5kdc[18120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:29:03.276446 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44709
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:03.358495 18137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:03.358739 18137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:03.358778 18137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:03.359913 18137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:03.359987 18137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:03.360010 18137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:03.360030 18137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:03.360061 18137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:03.361552 18137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:03.361826 18137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:03.362054 18137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:03.367156 18143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:03.367153 18144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:03.367177 18137 server_base.cc:1034] running on GCE node
W20260810 07:29:03.367156 18146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:03.367667 18137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:03.367908 18137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:03.369074 18137 hybrid_clock.cc:648] HybridClock initialized: now 1786346943369051 us; error 37 us; skew 500 ppm
Aug 10 07:29:03 dist-test-slave-vxj2 krb5kdc[18120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:03.377966 18137 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:29:03.378455 18137 webserver.cc:460] Webserver started at http://127.24.60.254:34705/ using document root <none> and password file <none>
I20260810 07:29:03.378729 18137 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:03.378803 18137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:03.378952 18137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:03.380218 18137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "2caab0e9832a4be5af2b21b88569d6ee"
format_stamp: "Formatted at 2026-08-10 07:29:03 on dist-test-slave-vxj2"
server_key: "b5114bcee383138ba64966696ee31a85"
server_key_iv: "9dd615bf0d50c1aae2aa6857a28b4a3d"
server_key_version: "encryptionkey@0"
I20260810 07:29:03.380712 18137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "2caab0e9832a4be5af2b21b88569d6ee"
format_stamp: "Formatted at 2026-08-10 07:29:03 on dist-test-slave-vxj2"
server_key: "b5114bcee383138ba64966696ee31a85"
server_key_iv: "9dd615bf0d50c1aae2aa6857a28b4a3d"
server_key_version: "encryptionkey@0"
I20260810 07:29:03.382936 18137 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 07:29:03.384083 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:03.384721 18137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:03.384828 18137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "2caab0e9832a4be5af2b21b88569d6ee"
format_stamp: "Formatted at 2026-08-10 07:29:03 on dist-test-slave-vxj2"
server_key: "b5114bcee383138ba64966696ee31a85"
server_key_iv: "9dd615bf0d50c1aae2aa6857a28b4a3d"
server_key_version: "encryptionkey@0"
I20260810 07:29:03.384917 18137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:03.418627 18137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:03.421192 18137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:03.421314 18137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:03.425446 18137 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:44709
I20260810 07:29:03.425490 18209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:44709 every 8 connection(s)
I20260810 07:29:03.425876 18137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:03.426817 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:03.429174 18210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee: Bootstrap starting.
I20260810 07:29:03.429735 18210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:03.429971 18210 log.cc:824] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:03.430687 18210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee: No bootstrap required, opened a new log
I20260810 07:29:03.432294 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18137
I20260810 07:29:03.432413 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:29:03.432418 18210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caab0e9832a4be5af2b21b88569d6ee" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44709 } }
I20260810 07:29:03.432629 24819 external_mini_cluster.cc:1428] Setting key 9f3b61e4c9a939a18c634c4344c930af
I20260810 07:29:03.432623 18210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:03.432667 18210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caab0e9832a4be5af2b21b88569d6ee, State: Initialized, Role: FOLLOWER
I20260810 07:29:03.432862 18210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caab0e9832a4be5af2b21b88569d6ee" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44709 } }
I20260810 07:29:03.432967 18210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:03.433008 18210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:03.433053 18210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:03.434059 18210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caab0e9832a4be5af2b21b88569d6ee" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44709 } }
I20260810 07:29:03.434190 18210 leader_election.cc:302] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2caab0e9832a4be5af2b21b88569d6ee; no voters:
I20260810 07:29:03.434366 18210 leader_election.cc:288] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:03.434414 18216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:03.434558 18216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [term 1 LEADER]: Becoming Leader. State: Replica: 2caab0e9832a4be5af2b21b88569d6ee, State: Running, Role: LEADER
I20260810 07:29:03.434649 18210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:03.434648 18216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caab0e9832a4be5af2b21b88569d6ee" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44709 } }
I20260810 07:29:03.435027 18217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2caab0e9832a4be5af2b21b88569d6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caab0e9832a4be5af2b21b88569d6ee" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44709 } } }
I20260810 07:29:03.435032 18218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2caab0e9832a4be5af2b21b88569d6ee. Latest consensus state: current_term: 1 leader_uuid: "2caab0e9832a4be5af2b21b88569d6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caab0e9832a4be5af2b21b88569d6ee" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 44709 } } }
I20260810 07:29:03.435207 18218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:03.435177 18217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:03.435602 18224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:03.436136 18224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:03.438958 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:03.433189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34265 (local address 127.24.60.254:44709)
0810 07:29:03.433376 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:03.433386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:03.433435 (+ 49us) server_negotiation.cc:407] Connection header received
0810 07:29:03.433572 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:03.433578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:03.433623 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:03.433765 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:03.434043 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:03.435472 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:03.436577 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:03.437394 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:03.437541 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:03.438292 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:03.438296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:03.438297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:03.438500 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:03.438532 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:03.438544 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:03.438618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:03.438709 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 07:29:03.438777 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260810 07:29:03.439941 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:03.441606 18224 catalog_manager.cc:1347] Generated new cluster ID: 47b8fee47e0b4d92b81b03dceb78bfd7
I20260810 07:29:03.441665 18224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:03.448822 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:03.441310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46213 (local address 127.24.60.254:44709)
0810 07:29:03.441351 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:03.441353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:03.445359 (+ 4006us) server_negotiation.cc:407] Connection header received
0810 07:29:03.445423 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:03.445424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:03.445436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:03.445495 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:03.445639 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:03.446374 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:03.447440 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:03.447734 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:03.447796 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:03.448559 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:03.448560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:03.448561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:03.448567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:03.448582 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:03.448587 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:03.448641 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:03.448702 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 07:29:03.448737 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":20}
I20260810 07:29:03.449666 18224 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:03.450165 18224 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:03.454206 18224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2caab0e9832a4be5af2b21b88569d6ee: Generated new TSK 0
I20260810 07:29:03.454388 18224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:03.461022 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18137
2026-08-10T07:29:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:03 dist-test-slave-vxj2 krb5kdc[18244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:03 dist-test-slave-vxj2 krb5kdc[18244](info): set up 2 sockets
Aug 10 07:29:03 dist-test-slave-vxj2 krb5kdc[18244](info): commencing operation
krb5kdc: starting...
W20260810 07:29:04.607350 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:29:04 dist-test-slave-vxj2 krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:29:04.736593 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34311
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:34311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:04.819970 18257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:04.820235 18257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:04.820274 18257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:04.821511 18257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:04.821587 18257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:04.821611 18257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:04.821633 18257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:04.821661 18257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:04.823173 18257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:34311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:34311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:04.823458 18257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:04.823698 18257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:04.828840 18265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:04.828836 18263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:04.828848 18262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:04.829088 18257 server_base.cc:1034] running on GCE node
I20260810 07:29:04.829471 18257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:04.829805 18257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:04.831018 18257 hybrid_clock.cc:648] HybridClock initialized: now 1786346944830992 us; error 37 us; skew 500 ppm
Aug 10 07:29:04 dist-test-slave-vxj2 krb5kdc[18244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:04.833999 18257 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:29:04.834501 18257 webserver.cc:460] Webserver started at http://127.24.60.254:37003/ using document root <none> and password file <none>
I20260810 07:29:04.834779 18257 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:04.834856 18257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:04.835024 18257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:04.836198 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "ee4499588c164a329fd9bf3ea5123c1d"
format_stamp: "Formatted at 2026-08-10 07:29:04 on dist-test-slave-vxj2"
server_key: "b98bf9049734acf054785ffc7aa8027a"
server_key_iv: "c59ca0d49e21094e957a956531a012fb"
server_key_version: "encryptionkey@0"
I20260810 07:29:04.836608 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "ee4499588c164a329fd9bf3ea5123c1d"
format_stamp: "Formatted at 2026-08-10 07:29:04 on dist-test-slave-vxj2"
server_key: "b98bf9049734acf054785ffc7aa8027a"
server_key_iv: "c59ca0d49e21094e957a956531a012fb"
server_key_version: "encryptionkey@0"
I20260810 07:29:04.838176 18257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:29:04.839072 18273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:04.839304 18257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:04.839449 18257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "ee4499588c164a329fd9bf3ea5123c1d"
format_stamp: "Formatted at 2026-08-10 07:29:04 on dist-test-slave-vxj2"
server_key: "b98bf9049734acf054785ffc7aa8027a"
server_key_iv: "c59ca0d49e21094e957a956531a012fb"
server_key_version: "encryptionkey@0"
I20260810 07:29:04.839532 18257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:04.843011 18257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:04.845422 18257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:04.845592 18257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:04.849532 18257 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:34311
I20260810 07:29:04.849562 18335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:34311 every 8 connection(s)
I20260810 07:29:04.849891 18257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:04.850885 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:04.852253 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18257
I20260810 07:29:04.852409 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:29:04.852610 24819 external_mini_cluster.cc:1428] Setting key 93a1d32ebd1e86da7e5275d650822850
I20260810 07:29:04.853772 18336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d: Bootstrap starting.
I20260810 07:29:04.854566 18336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:04.854863 18336 log.cc:824] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:04.855608 18336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d: No bootstrap required, opened a new log
I20260810 07:29:04.856455 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:04.853204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53305 (local address 127.24.60.254:34311)
0810 07:29:04.854546 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:04.854554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:04.854580 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:29:04.855128 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:04.855136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:04.855164 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:04.855332 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:04.856006 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:04.856011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:04.856011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:04.856016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:04.856064 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:04.856078 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:04.856177 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:04.856208 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 07:29:04.856248 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":73,"threads_started":1}
I20260810 07:29:04.857599 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:04.857952 18336 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4499588c164a329fd9bf3ea5123c1d" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34311 } }
I20260810 07:29:04.858084 18336 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:04.858106 18336 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4499588c164a329fd9bf3ea5123c1d, State: Initialized, Role: FOLLOWER
I20260810 07:29:04.858243 18336 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4499588c164a329fd9bf3ea5123c1d" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34311 } }
I20260810 07:29:04.858335 18336 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:04.858356 18336 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:04.858403 18336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:04.859082 18336 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4499588c164a329fd9bf3ea5123c1d" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34311 } }
I20260810 07:29:04.859218 18336 leader_election.cc:302] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee4499588c164a329fd9bf3ea5123c1d; no voters:
I20260810 07:29:04.859388 18336 leader_election.cc:288] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:04.859426 18346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:04.859660 18346 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [term 1 LEADER]: Becoming Leader. State: Replica: ee4499588c164a329fd9bf3ea5123c1d, State: Running, Role: LEADER
I20260810 07:29:04.859771 18336 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:04.859802 18346 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4499588c164a329fd9bf3ea5123c1d" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34311 } }
I20260810 07:29:04.860108 18346 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4499588c164a329fd9bf3ea5123c1d. Latest consensus state: current_term: 1 leader_uuid: "ee4499588c164a329fd9bf3ea5123c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4499588c164a329fd9bf3ea5123c1d" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34311 } } }
I20260810 07:29:04.860189 18346 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:04.860440 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:04.859257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57145 (local address 127.24.60.254:34311)
0810 07:29:04.859297 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:04.859299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:04.859370 (+ 71us) server_negotiation.cc:407] Connection header received
0810 07:29:04.859452 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:04.859453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:04.859457 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:04.859518 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:04.860181 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:04.860182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:04.860183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:04.860188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:04.860203 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:04.860208 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:04.860242 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:04.860366 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 07:29:04.860381 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:29:04.860118 18347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee4499588c164a329fd9bf3ea5123c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4499588c164a329fd9bf3ea5123c1d" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 34311 } } }
I20260810 07:29:04.861372 18347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:04.861413 18351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:04.861898 18351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:04.863416 18351 catalog_manager.cc:1347] Generated new cluster ID: 3c129f63bad548218392cd50710e6791
I20260810 07:29:04.863479 18351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:04.880031 18351 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:04.880237 18351 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:04.882613 18351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee4499588c164a329fd9bf3ea5123c1d: Generated new TSK 0
I20260810 07:29:04.882797 18351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:04.885011 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18257
2026-08-10T07:29:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1422 ms)
[----------] 18 tests from AuthTokenIssuingTest (25899 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:04 dist-test-slave-vxj2 krb5kdc[18373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:04 dist-test-slave-vxj2 krb5kdc[18373](info): set up 2 sockets
Aug 10 07:29:04 dist-test-slave-vxj2 krb5kdc[18373](info): commencing operation
krb5kdc: starting...
W20260810 07:29:06.015164 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:29:06.144348 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:40817
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:40817,127.24.60.253:40665,127.24.60.252:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:06.230194 18386 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:06.230471 18386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:06.230505 18386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:06.230553 18386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:06.231725 18386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:06.231812 18386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:06.231837 18386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:06.231868 18386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:06.231899 18386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:06.233520 18386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:40817,127.24.60.253:40665,127.24.60.252:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:06.233843 18386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:06.234098 18386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:06.239284 18395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:06.239259 18392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:06.239259 18393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:06.239507 18386 server_base.cc:1034] running on GCE node
I20260810 07:29:06.239928 18386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:06.240249 18386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:06.241446 18386 hybrid_clock.cc:648] HybridClock initialized: now 1786346946241432 us; error 61 us; skew 500 ppm
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:06.244417 18386 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:29:06.244948 18386 webserver.cc:460] Webserver started at http://127.24.60.254:46755/ using document root <none> and password file <none>
I20260810 07:29:06.245261 18386 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:06.245344 18386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:06.245486 18386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:06.246666 18386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "480de86898e1460fb13c5919ad247f27"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "9cbe1ec1f65564a4515beb275499da1e"
server_key_iv: "54909c339e34d8150602a669b7f9d26d"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.247074 18386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "480de86898e1460fb13c5919ad247f27"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "9cbe1ec1f65564a4515beb275499da1e"
server_key_iv: "54909c339e34d8150602a669b7f9d26d"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.248459 18386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:29:06.249357 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:06.249689 18386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:06.249792 18386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "480de86898e1460fb13c5919ad247f27"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "9cbe1ec1f65564a4515beb275499da1e"
server_key_iv: "54909c339e34d8150602a669b7f9d26d"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.249931 18386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:06.282974 18386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:06.285130 18386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:06.285355 18386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:06.289539 18386 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:40817
I20260810 07:29:06.289574 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:40817 every 8 connection(s)
I20260810 07:29:06.289954 18386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:06.290522 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18386
I20260810 07:29:06.290660 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:29:06.290899 24819 external_mini_cluster.cc:1428] Setting key b69434ebdc7f4e8e7b71c10d7eb3f034
I20260810 07:29:06.291291 18464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:06.293506 18464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.60.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.253@KRBTEST.COM" created.
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:06.305498 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.294298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:40817 (local address 127.0.0.1:56710)
0810 07:29:06.294493 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.294528 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.295030 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.295327 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.295329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.295616 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.295772 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.295776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.296822 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.296823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.297563 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.297566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.298090 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.298091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.298156 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.298725 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.298742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.302135 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.303961 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.303967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.303969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.304167 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.304428 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.304429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.304429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.304520 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.304878 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.304883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.304953 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.305063 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.305311 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":110,"thread_start_us":93,"threads_started":1}
I20260810 07:29:06.305500 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.294516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.24.60.254:40817)
0810 07:29:06.294644 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.294649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.294681 (+ 32us) server_negotiation.cc:407] Connection header received
0810 07:29:06.295199 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.295202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.295219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.295322 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.295883 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.296705 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.297661 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.297969 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.298102 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.302239 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.302247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.302247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.302261 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.303849 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.304272 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.304275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.304276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.304357 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.304587 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.304588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.304589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.304804 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.304901 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.305097 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.305253 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260810 07:29:06.306596 18464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:06.306981 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.306853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56886)
0810 07:29:06.306904 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.306961 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 07:29:06.307107 18405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
I20260810 07:29:06.307529 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.307430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56898)
0810 07:29:06.307482 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.307507 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 07:29:06.307637 18464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:40665: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab.
Entry for principal kudu/127.24.60.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab.
I20260810 07:29:06.316630 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:40665
--webserver_interface=127.24.60.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab
--principal=kudu/127.24.60.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:40817,127.24.60.253:40665,127.24.60.252:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:06.356870 18464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } attempt: 1
I20260810 07:29:06.357501 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.357358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56912)
0810 07:29:06.357426 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.357471 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 07:29:06.357908 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.357818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56920)
0810 07:29:06.357864 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.357891 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 07:29:06.358043 18464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:40665: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
W20260810 07:29:06.404138 18473 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:06.404402 18473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:06.404423 18473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:06.404448 18473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:06.405882 18473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:06.405939 18473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:06.405956 18473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:06.405969 18473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:06.406025 18473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:06.407809 18473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:40817,127.24.60.253:40665,127.24.60.252:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab
--principal=kudu/127.24.60.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.60.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:06.408110 18473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:06.408372 18473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:29:06.414121 18473 server_base.cc:1034] running on GCE node
W20260810 07:29:06.414126 18480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:06.414084 18479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:06.414090 18482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:06.414525 18473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:06.414796 18473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:06.415946 18473 hybrid_clock.cc:648] HybridClock initialized: now 1786346946415926 us; error 44 us; skew 500 ppm
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:06.418608 18473 init.cc:377] Logged in from keytab as kudu/127.24.60.253@KRBTEST.COM (short username kudu)
I20260810 07:29:06.419100 18473 webserver.cc:460] Webserver started at http://127.24.60.253:34407/ using document root <none> and password file <none>
I20260810 07:29:06.419348 18473 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:06.419396 18473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:06.419565 18473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:06.420683 18473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data/instance:
uuid: "5f37797cc311485389ec2dd7f05ea997"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "aa04691b3b2577c9369eb690d783e0d9"
server_key_iv: "10685e2597cfb8133f4f43bf2690985a"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.421063 18473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal/instance:
uuid: "5f37797cc311485389ec2dd7f05ea997"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "aa04691b3b2577c9369eb690d783e0d9"
server_key_iv: "10685e2597cfb8133f4f43bf2690985a"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.422534 18473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:29:06.423382 18489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:06.423578 18473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:29:06.423658 18473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal
uuid: "5f37797cc311485389ec2dd7f05ea997"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "aa04691b3b2577c9369eb690d783e0d9"
server_key_iv: "10685e2597cfb8133f4f43bf2690985a"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.423751 18473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:06.426208 18464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } attempt: 2
I20260810 07:29:06.426694 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.426554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56926)
0810 07:29:06.426624 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.426664 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 07:29:06.427101 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.427001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56936)
0810 07:29:06.427048 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.427082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 07:29:06.427201 18464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:40665: Network error: Client connection negotiation failed: client connection to 127.24.60.253:40665: connect: Connection refused (error 111)
I20260810 07:29:06.433326 18473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:06.435392 18473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:06.435559 18473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:06.439558 18473 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.253:40665
I20260810 07:29:06.439631 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.253:40665 every 8 connection(s)
I20260810 07:29:06.439977 18473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
I20260810 07:29:06.441020 18554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:06.442229 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18473
I20260810 07:29:06.442339 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-1/wal/instance
I20260810 07:29:06.442538 24819 external_mini_cluster.cc:1428] Setting key 802e4331110f5de31cb49cbafda9caf3
I20260810 07:29:06.443097 18554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.60.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.252@KRBTEST.COM" created.
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:06.457939 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.443964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.24.60.254:40817)
0810 07:29:06.444039 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.444042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.444259 (+ 217us) server_negotiation.cc:407] Connection header received
0810 07:29:06.444768 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.444770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.444785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.444843 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.446812 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.447667 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.448707 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.449058 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.449143 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.454686 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.454688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.454688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.454697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.456241 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.456751 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.456753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.456753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.456822 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.457111 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.457112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.457113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.457302 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.457375 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.457684 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.457790 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:29:06.457969 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.443932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:40817 (local address 127.0.0.1:56722)
0810 07:29:06.444116 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.444157 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.444659 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.444915 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.444918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.446470 (+ 1552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.446697 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.446702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.447777 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.447778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.448608 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.448611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.449221 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.449223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.449342 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.449903 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.449912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.454544 (+ 4632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.456364 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.456376 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.456379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.456651 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.456897 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.456898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.456898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.457033 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.457373 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.457380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.457536 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.457635 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.457784 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260810 07:29:06.458678 18554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:29:06.469964 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.459016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56952)
0810 07:29:06.459080 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.459093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.459141 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.459393 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.459394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.459587 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.459662 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.459664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.460553 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.460555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.461416 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.461419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.461893 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.461894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.461975 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.462407 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.462414 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.466083 (+ 3669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.468527 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.468528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.468528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.468781 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.469108 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.469108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.469109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.469198 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.469625 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.469625 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.469669 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.469728 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.469838 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:29:06.470017 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.459057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.24.60.253:40665)
0810 07:29:06.459232 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.459237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.459256 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:29:06.459285 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.459287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.459299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.459364 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.459747 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.460457 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.461506 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.461809 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.461867 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.466220 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.466231 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.466231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.466248 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.468431 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.468940 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.468944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.468945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.469022 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.469252 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.469253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.469268 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.469539 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.469651 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.469771 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.469894 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260810 07:29:06.470671 18554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:06.471102 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.470973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46322)
0810 07:29:06.471026 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.471074 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 07:29:06.471262 18492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
I20260810 07:29:06.471750 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.471665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46336)
0810 07:29:06.471708 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.471733 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 07:29:06.471928 18554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:35639: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab.
Entry for principal kudu/127.24.60.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab.
I20260810 07:29:06.473011 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:35639
--webserver_interface=127.24.60.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab
--principal=kudu/127.24.60.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:40817,127.24.60.253:40665,127.24.60.252:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:06.518391 18464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } attempt: 3
I20260810 07:29:06.521194 18554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } attempt: 1
I20260810 07:29:06.521608 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.521469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46352)
0810 07:29:06.521529 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.521567 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:29:06.522495 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.522384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46360)
0810 07:29:06.522430 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.522464 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 07:29:06.522614 18554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:35639: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:29:06.525739 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.518899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.24.60.253:40665)
0810 07:29:06.518988 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.518990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.519001 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:29:06.519041 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.519041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.519053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.519118 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.519460 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.519996 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.520702 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.520931 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.521016 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.523310 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.523311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.523312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.523318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.524644 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.525037 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.525038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.525038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.525091 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.525303 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.525304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.525304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.525413 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.525469 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.525623 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.525684 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:29:06.525883 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.518790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56968)
0810 07:29:06.518862 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.518877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.518946 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.519126 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.519127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.519303 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.519372 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.519374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.520090 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.520091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.520603 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.520604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.521016 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.521017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.521127 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.521713 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.521718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.523207 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.524727 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.524728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.524729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.524960 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.525172 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.525173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.525173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.525243 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.525476 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.525477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.525531 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.525608 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.525722 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 07:29:06.526346 18464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:06.526718 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.526599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46364)
0810 07:29:06.526653 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.526690 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 07:29:06.527086 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.526973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46370)
0810 07:29:06.527028 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.527060 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 07:29:06.527210 18464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:35639: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
I20260810 07:29:06.558382 18464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } attempt: 1
I20260810 07:29:06.558930 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.558753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46376)
0810 07:29:06.558820 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.558862 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 07:29:06.559358 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.559199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46378)
0810 07:29:06.559269 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.559317 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 07:29:06.559623 18464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:35639: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
W20260810 07:29:06.560601 18562 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:06.560818 18562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:06.560837 18562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:06.560858 18562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:06.562079 18562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:06.562124 18562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:06.562137 18562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:06.562151 18562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:06.562163 18562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:06.563817 18562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:40817,127.24.60.253:40665,127.24.60.252:35639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab
--principal=kudu/127.24.60.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.60.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:06.564036 18562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:06.564241 18562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:06.569459 18568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:06.569485 18571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:06.569468 18569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:06.569782 18562 server_base.cc:1034] running on GCE node
I20260810 07:29:06.570109 18562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:06.570429 18562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:06.571604 18562 hybrid_clock.cc:648] HybridClock initialized: now 1786346946571584 us; error 40 us; skew 500 ppm
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:06.574311 18562 init.cc:377] Logged in from keytab as kudu/127.24.60.252@KRBTEST.COM (short username kudu)
I20260810 07:29:06.574792 18562 webserver.cc:460] Webserver started at http://127.24.60.252:40981/ using document root <none> and password file <none>
I20260810 07:29:06.574990 18562 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:06.575031 18562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:06.575124 18562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:06.576135 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data/instance:
uuid: "32c6cf5112b2427594f44deda0e67123"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "d16c77d32cb6cb075b2440bd62ff6108"
server_key_iv: "3ab2aaa0cc933902b5a83b45ef9d0a30"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.576445 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal/instance:
uuid: "32c6cf5112b2427594f44deda0e67123"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "d16c77d32cb6cb075b2440bd62ff6108"
server_key_iv: "3ab2aaa0cc933902b5a83b45ef9d0a30"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.577905 18562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:29:06.578667 18578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:06.578907 18562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:06.578987 18562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal
uuid: "32c6cf5112b2427594f44deda0e67123"
format_stamp: "Formatted at 2026-08-10 07:29:06 on dist-test-slave-vxj2"
server_key: "d16c77d32cb6cb075b2440bd62ff6108"
server_key_iv: "3ab2aaa0cc933902b5a83b45ef9d0a30"
server_key_version: "encryptionkey@0"
I20260810 07:29:06.579097 18562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:06.590762 18554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } attempt: 2
I20260810 07:29:06.591400 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.591213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46390)
0810 07:29:06.591320 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.591366 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 07:29:06.591924 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.591827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46400)
0810 07:29:06.591875 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.591905 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 07:29:06.592067 18554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:35639: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
I20260810 07:29:06.628371 18562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:06.630887 18562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:06.631067 18562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:06.634800 18464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } attempt: 2
I20260810 07:29:06.635340 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.635165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46416)
0810 07:29:06.635238 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.635287 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 07:29:06.635635 18562 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.252:35639
I20260810 07:29:06.635720 18639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.252:35639 every 8 connection(s)
I20260810 07:29:06.636094 18562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
I20260810 07:29:06.636857 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:06.639008 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18562
I20260810 07:29:06.639134 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346663269703-24819-0/minicluster-data/master-2/wal/instance
I20260810 07:29:06.639320 24819 external_mini_cluster.cc:1428] Setting key fb465df9069ce12d710e6a9748d54b22
I20260810 07:29:06.639101 18640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:29:06.648633 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.635744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46426)
0810 07:29:06.635788 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.635797 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.635840 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.637767 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.637768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.637980 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.638038 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.638040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.639641 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.639643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.640480 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.640485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.641594 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.641595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.641657 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.642194 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.642200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.643836 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.647137 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.647139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.647139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.647417 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.647681 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.647682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.647682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.647751 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.648158 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.648159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.648213 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.648369 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.648492 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 07:29:06.649403 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.636096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.24.60.252:35639)
0810 07:29:06.636807 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.636817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.636861 (+ 44us) server_negotiation.cc:407] Connection header received
0810 07:29:06.637528 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.637537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.637604 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.637770 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.638159 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.639545 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.640580 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.641398 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.641502 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.643938 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.643947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.643948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.643955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.647043 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.647497 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.647500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.647501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.647599 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.647810 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.647811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.647812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.648070 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.648165 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.648915 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.649084 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":659,"thread_start_us":604,"threads_started":1}
I20260810 07:29:06.650350 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.640325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.24.60.254:40817)
0810 07:29:06.640378 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.640380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.640388 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:29:06.640415 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.640416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.640424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.640490 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.641030 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.641615 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.642655 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.642978 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.643057 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.646489 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.646491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.646491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.646499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.648239 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.648731 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.648732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.648733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.648811 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.649055 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.649056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.649057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.649258 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.649352 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.650162 (+ 810us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.650258 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:29:06.651314 18464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27: Bootstrap starting.
I20260810 07:29:06.652182 18464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:06.652523 18464 log.cc:824] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:06.653107 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.639812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:40817 (local address 127.0.0.1:56738)
0810 07:29:06.640132 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.640180 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.640238 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.640539 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.640542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.640830 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.640926 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.640928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.641716 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.641718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.642537 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.642539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.643094 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.643095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.643174 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.643727 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.643739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.646390 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.648365 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.648374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.648377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.648620 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.648895 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.648896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.648896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.648977 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.649343 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.649351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.649413 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.650091 (+ 678us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.652933 (+ 2842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260810 07:29:06.653527 18464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27: No bootstrap required, opened a new log
I20260810 07:29:06.655061 18464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.655206 18464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480de86898e1460fb13c5919ad247f27, State: Initialized, Role: FOLLOWER
I20260810 07:29:06.655342 18464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.655648 18464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:06.655611 18647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.655747 18647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.656946 18640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } has no permanent_uuid. Determining permanent_uuid...
W20260810 07:29:06.657400 18659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:06.657469 18659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:06.658986 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.639987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.24.60.252:35639)
0810 07:29:06.640124 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.640126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.640135 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:29:06.640159 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.640160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.640171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.640253 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.641082 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.641806 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.642839 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.643134 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.643194 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.647163 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.647164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.647164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.647171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.652990 (+ 5819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.657582 (+ 4592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.657583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.657583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.657656 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.657919 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.657920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.657920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.658066 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.658715 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.658758 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.658849 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260810 07:29:06.658460 18386 master_runner.cc:186] Error getting master registration for 127.24.60.253:40665: OK, instance_id { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" instance_seqno: 1786346946438443 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:06.661703 18386 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:06.665403 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.657286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:40665 (local address 127.0.0.1:56972)
0810 07:29:06.657396 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.657407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.657446 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.657756 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.657757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.657929 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.657997 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.657999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.658925 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.658926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.659702 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.659705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.660238 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.660239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.660309 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.660832 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.660837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.662372 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.664203 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.664204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.664205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.664424 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.664661 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.664662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.664662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.664731 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.665057 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.665058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.665104 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.665196 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.665303 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 07:29:06.665747 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.657526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.24.60.253:40665)
0810 07:29:06.657582 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.657585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.657595 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:29:06.657623 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.657624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.657635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.657702 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.658087 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.658823 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.659806 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.660111 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.660180 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.662470 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.662471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.662472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.662480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.664095 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.664510 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.664512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.664512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.664576 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.664806 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.664807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.664807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.664955 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.665026 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.665538 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.665656 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 07:29:06.666241 18640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:06.668426 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.659764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.24.60.254:40817)
0810 07:29:06.659804 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.659806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.659813 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:29:06.659832 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.659832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.659841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.659886 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.660545 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.661265 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.662280 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.662556 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.662629 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.665454 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.665456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.665456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.665463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.667194 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.667615 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.667616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.667617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.667680 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.667893 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.667894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.667895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.668034 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.668206 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.668252 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.668332 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:29:06.673890 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.666667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46438)
0810 07:29:06.666715 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.666725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.666763 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.666878 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.666878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.667013 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.667041 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.667042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.667686 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.667687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.668373 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.668376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.668856 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.668858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.668942 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.669347 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.669353 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.670731 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.672568 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.672569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.672570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.672788 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.673015 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.673016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.673016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.673085 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.673515 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.673516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.673563 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.673641 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.673753 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 07:29:06.674090 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.666704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.24.60.252:35639)
0810 07:29:06.666737 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.666739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.666750 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:29:06.666809 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.666810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.666819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.666874 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.667111 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.667592 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.668457 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.668734 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.668847 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.670820 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.670828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.670828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.670835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.672471 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.672874 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.672875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.672875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.672941 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.673214 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.673215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.673215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.673435 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.673509 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.673689 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.673830 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":17}
I20260810 07:29:06.675601 18640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: Bootstrap starting.
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:29:06.676342 18640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:06.676692 18640 log.cc:824] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:06.677791 18640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: No bootstrap required, opened a new log
I20260810 07:29:06.679545 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.669362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.24.60.253:40665)
0810 07:29:06.669424 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.669427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.669474 (+ 47us) server_negotiation.cc:407] Connection header received
0810 07:29:06.669550 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.669551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.669560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.669609 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.670083 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.670592 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.672104 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.672910 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.672980 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.676200 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.676202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.676202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.676210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.678157 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.678578 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.678580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.678580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.678647 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.678978 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.678979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.678979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.679127 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.679313 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.679367 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.679453 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:29:06.680305 18640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.680487 18640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c6cf5112b2427594f44deda0e67123, State: Initialized, Role: FOLLOWER
I20260810 07:29:06.680583 18640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.680907 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:06.680974 18640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:06.680922 18662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.681020 18662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.682190 18431 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "480de86898e1460fb13c5919ad247f27"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740
I20260810 07:29:06.682400 18431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 07:29:06.682469 18431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:06.683221 18554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } attempt: 3
I20260810 07:29:06.683606 18431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.683892 18431 leader_election.cc:288] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [CANDIDATE]: Term 1 election: Requested vote from peers 5f37797cc311485389ec2dd7f05ea997 (127.24.60.253:40665), 32c6cf5112b2427594f44deda0e67123 (127.24.60.252:35639)
I20260810 07:29:06.684360 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "480de86898e1460fb13c5919ad247f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "32c6cf5112b2427594f44deda0e67123"
I20260810 07:29:06.684480 18606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:06.685384 18606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480de86898e1460fb13c5919ad247f27 in term 1.
I20260810 07:29:06.685673 18405 leader_election.cc:302] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c6cf5112b2427594f44deda0e67123, 480de86898e1460fb13c5919ad247f27; no voters:
I20260810 07:29:06.685807 18647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:06.686036 18647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [term 1 LEADER]: Becoming Leader. State: Replica: 480de86898e1460fb13c5919ad247f27, State: Running, Role: LEADER
I20260810 07:29:06.686137 18647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.686626 18647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480de86898e1460fb13c5919ad247f27. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.686697 18647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: This master's current role is: LEADER
W20260810 07:29:06.687166 18674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:06.687237 18674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 07:29:06 dist-test-slave-vxj2 krb5kdc[18373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346946, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:29:06.686477 18562 master_runner.cc:186] Error getting master registration for 127.24.60.253:40665: OK, instance_id { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" instance_seqno: 1786346946438443 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:06.688803 18562 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:29:06.690171 18682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:06.690956 18682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:06.691587 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.683508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:35639 (local address 127.0.0.1:46448)
0810 07:29:06.683557 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.683570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:06.683619 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:06.683776 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:06.683777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:06.683968 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:06.684016 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.684019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.684951 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.684952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.685732 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:06.685735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.686222 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.686223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:06.686284 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.686885 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:06.686891 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:06.688434 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:06.690439 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.690440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.690440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.690670 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.690900 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:06.690901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:06.690901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:06.690967 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:06.691263 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:06.691264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:06.691314 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 07:29:06.691381 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:29:06.691490 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 07:29:06.691954 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.683563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.24.60.252:35639)
0810 07:29:06.683611 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.683614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.683624 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:29:06.683669 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.683670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.683680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.683727 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.684178 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.684853 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.685841 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.686102 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.686168 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.688532 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.688533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.688534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.688542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.690310 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.690755 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.690756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.690756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.690815 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.691036 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.691038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.691038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.691170 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.691233 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.691770 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.691867 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:29:06.693528 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: Bootstrap starting.
I20260810 07:29:06.694326 18554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:06.694664 18554 log.cc:824] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:06.695647 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: No bootstrap required, opened a new log
I20260810 07:29:06.697861 18554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.698028 18554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f37797cc311485389ec2dd7f05ea997, State: Initialized, Role: FOLLOWER
I20260810 07:29:06.698114 18554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } }
I20260810 07:29:06.698467 18554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:06.698498 18520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "480de86898e1460fb13c5919ad247f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f37797cc311485389ec2dd7f05ea997"
I20260810 07:29:06.698602 18520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:06.699613 18520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 480de86898e1460fb13c5919ad247f27 in term 1.
I20260810 07:29:06.702698 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.689562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.24.60.254:40817)
0810 07:29:06.689622 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.689624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.693829 (+ 4205us) server_negotiation.cc:407] Connection header received
0810 07:29:06.694120 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.694121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.694134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.694200 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.695150 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.695946 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.697066 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.697383 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.697427 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.699402 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.699403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.699403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.699412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.701282 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.701705 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.701706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.701707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.701772 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.702009 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.702010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.702010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.702149 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.702231 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.702500 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.702588 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 07:29:06.704262 18701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:06.704324 18701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:06.705240 18690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.705335 18690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.709231 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.689746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.24.60.253:40665)
0810 07:29:06.689800 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.689802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.700172 (+ 10370us) server_negotiation.cc:407] Connection header received
0810 07:29:06.700270 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.700271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.700282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.700312 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.700958 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.701697 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.702808 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.703113 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.703159 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.706079 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.706080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.706081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.706090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.707867 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.708301 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.708302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.708302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.708364 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.708572 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.708573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.708573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.708707 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.708792 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.709023 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.709107 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 07:29:06.710494 18606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [term 1 FOLLOWER]: Refusing update from remote peer 480de86898e1460fb13c5919ad247f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:06.710850 18680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:06.711519 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.689950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.24.60.252:35639)
0810 07:29:06.690407 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.690408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.703354 (+ 12946us) server_negotiation.cc:407] Connection header received
0810 07:29:06.703463 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.703464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.703474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.703505 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.704160 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.704885 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.706111 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:06.706428 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:06.706479 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:06.708087 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:06.708089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:06.708089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:06.708097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:06.709786 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.710426 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.710427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.710428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.710499 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:06.710782 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:06.710782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:06.710783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:06.710917 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:06.710991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:06.711317 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 07:29:06.711409 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260810 07:29:06.712352 18662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480de86898e1460fb13c5919ad247f27. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.712445 18662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.712914 18520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [term 1 FOLLOWER]: Refusing update from remote peer 480de86898e1460fb13c5919ad247f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:06.713261 18680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:06.714480 18647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.714570 18647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:06.714709 18662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.714808 18662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.714913 18688 mvcc.cc:205] Tried to move back new op lower bound from 7316877093722894336 to 7316877093627273216. Current Snapshot: MvccSnapshot[applied={T|T < 7316877093722894336}]
I20260810 07:29:06.715014 18704 mvcc.cc:205] Tried to move back new op lower bound from 7316877093722894336 to 7316877093627273216. Current Snapshot: MvccSnapshot[applied={T|T < 7316877093722894336}]
I20260810 07:29:06.715457 18690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480de86898e1460fb13c5919ad247f27. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.715557 18690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.716140 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.716257 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:06.716377 18682 catalog_manager.cc:1347] Generated new cluster ID: c4680a6324704ac6bf94f2f1213e670d
I20260810 07:29:06.716418 18682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:06.716739 18690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "480de86898e1460fb13c5919ad247f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "480de86898e1460fb13c5919ad247f27" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 40817 } } peers { permanent_uuid: "5f37797cc311485389ec2dd7f05ea997" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 40665 } } peers { permanent_uuid: "32c6cf5112b2427594f44deda0e67123" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 35639 } } }
I20260810 07:29:06.716809 18690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:06.717276 18706 mvcc.cc:205] Tried to move back new op lower bound from 7316877093722894336 to 7316877093627273216. Current Snapshot: MvccSnapshot[applied={T|T < 7316877093722894336}]
I20260810 07:29:06.737115 18682 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:06.738112 18682 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:06.741801 18682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 480de86898e1460fb13c5919ad247f27: Generated new TSK 0
I20260810 07:29:06.742038 18682 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:29:06.774004 18470 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.772224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.24.60.254:40817)
0810 07:29:06.772273 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.772275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.773259 (+ 984us) server_negotiation.cc:407] Connection header received
0810 07:29:06.773375 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.773377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.773390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.773423 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.773922 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56756: BlockingRecv error: recv got EOF from 127.0.0.1:56756 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 07:29:06.774787 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.772633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.24.60.253:40665)
0810 07:29:06.772676 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.772678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.774260 (+ 1582us) server_negotiation.cc:407] Connection header received
0810 07:29:06.774361 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.774362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.774374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.774403 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.774723 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56996: BlockingRecv error: recv got EOF from 127.0.0.1:56996 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 07:29:06.775543 18642 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.772931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.24.60.252:35639)
0810 07:29:06.772974 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.772976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.775026 (+ 2050us) server_negotiation.cc:407] Connection header received
0810 07:29:06.775124 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.775125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.775135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.775167 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.775468 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46460: BlockingRecv error: recv got EOF from 127.0.0.1:46460 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 07:29:06.778620 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.777806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.24.60.253:40665)
0810 07:29:06.777856 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.777858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.778088 (+ 230us) server_negotiation.cc:407] Connection header received
0810 07:29:06.778198 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.778199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.778219 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.778248 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.778572 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56998: BlockingRecv error: recv got EOF from 127.0.0.1:56998 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 07:29:06.786041 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.781436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.24.60.253:40665)
0810 07:29:06.781472 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.781474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.785407 (+ 3933us) server_negotiation.cc:407] Connection header received
0810 07:29:06.785523 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.785525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.785537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.785569 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.785981 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57012: BlockingRecv error: recv got EOF from 127.0.0.1:57012 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 07:29:06.790216 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.789325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.24.60.253:40665)
0810 07:29:06.789366 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.789368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.789631 (+ 263us) server_negotiation.cc:407] Connection header received
0810 07:29:06.789734 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.789735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.789747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.789778 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.790156 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57014: BlockingRecv error: recv got EOF from 127.0.0.1:57014 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 07:29:06.796348 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.795497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.24.60.253:40665)
0810 07:29:06.795538 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.795539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.795793 (+ 254us) server_negotiation.cc:407] Connection header received
0810 07:29:06.795905 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.795906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.795918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.795948 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.796292 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57024: BlockingRecv error: recv got EOF from 127.0.0.1:57024 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 07:29:06.807863 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.806902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.24.60.253:40665)
0810 07:29:06.806947 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.806950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.807234 (+ 284us) server_negotiation.cc:407] Connection header received
0810 07:29:06.807342 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.807343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.807356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.807388 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.807785 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57034: BlockingRecv error: recv got EOF from 127.0.0.1:57034 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 07:29:06.826561 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.825447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.24.60.253:40665)
0810 07:29:06.825494 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.825496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.825797 (+ 301us) server_negotiation.cc:407] Connection header received
0810 07:29:06.825928 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.825930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.825943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.825978 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.826453 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57046: BlockingRecv error: recv got EOF from 127.0.0.1:57046 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 07:29:06.861331 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.860220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.24.60.253:40665)
0810 07:29:06.860280 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.860283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.860631 (+ 348us) server_negotiation.cc:407] Connection header received
0810 07:29:06.860682 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.860683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.860698 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.860778 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.861258 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57050: BlockingRecv error: recv got EOF from 127.0.0.1:57050 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 07:29:06.927439 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:06.926450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.24.60.253:40665)
0810 07:29:06.926551 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:06.926553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:06.926640 (+ 87us) server_negotiation.cc:407] Connection header received
0810 07:29:06.926721 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:06.926722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:06.926737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:06.926910 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:06.927349 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57058: BlockingRecv error: recv got EOF from 127.0.0.1:57058 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 07:29:07.057561 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:07.056771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.24.60.253:40665)
0810 07:29:07.056902 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.056905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.056949 (+ 44us) server_negotiation.cc:407] Connection header received
0810 07:29:07.057040 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.057041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.057051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.057107 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:07.057486 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57066: BlockingRecv error: recv got EOF from 127.0.0.1:57066 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260810 07:29:07.315817 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:07.315071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.24.60.253:40665)
0810 07:29:07.315150 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.315153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.315268 (+ 115us) server_negotiation.cc:407] Connection header received
0810 07:29:07.315297 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.315298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.315310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.315370 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:07.315753 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57078: BlockingRecv error: recv got EOF from 127.0.0.1:57078 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:29:07.687700 18674 catalog_manager.cc:1259] Loaded cluster ID: c4680a6324704ac6bf94f2f1213e670d
I20260810 07:29:07.687783 18674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: loading cluster ID for follower catalog manager: success
I20260810 07:29:07.688575 18674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: acquiring CA information for follower catalog manager: success
I20260810 07:29:07.688829 18674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32c6cf5112b2427594f44deda0e67123: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:07.704890 18701 catalog_manager.cc:1259] Loaded cluster ID: c4680a6324704ac6bf94f2f1213e670d
I20260810 07:29:07.704974 18701 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: loading cluster ID for follower catalog manager: success
I20260810 07:29:07.705838 18701 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: acquiring CA information for follower catalog manager: success
I20260810 07:29:07.706120 18701 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f37797cc311485389ec2dd7f05ea997: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:07.831833 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.828979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.24.60.253:40665)
0810 07:29:07.829110 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.829113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.829202 (+ 89us) server_negotiation.cc:407] Connection header received
0810 07:29:07.829346 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.829347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.829400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.829493 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:07.829651 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.830153 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.830983 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.831189 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.831266 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:07.831463 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:07.831574 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:07.831668 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:07.831707 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 07:29:07.831729 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260810 07:29:07.836087 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.833839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.24.60.252:35639)
0810 07:29:07.833941 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.833943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.833950 (+ 7us) server_negotiation.cc:407] Connection header received
0810 07:29:07.833987 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.833988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.834022 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.834109 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:07.834220 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.834678 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.835382 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.835577 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.835632 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:07.835754 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:07.835904 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:07.835967 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:07.836006 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 07:29:07.836025 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260810 07:29:07.840968 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.838153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.24.60.253:40665)
0810 07:29:07.838223 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.838225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.838379 (+ 154us) server_negotiation.cc:407] Connection header received
0810 07:29:07.838513 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.838514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.838526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.838558 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:07.838741 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.839241 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.840134 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.840432 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.840503 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:07.840652 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:07.840692 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:07.840747 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:07.840871 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 07:29:07.840897 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:29:07.841367 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.838002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.24.60.254:40817)
0810 07:29:07.838123 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.838125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.838135 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:29:07.838183 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.838183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.838236 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.838339 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:07.838531 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.839163 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.840289 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.840597 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.840679 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:07.840927 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:07.841039 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:07.841092 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:07.841243 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 07:29:07.841269 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":79,"threads_started":1}
I20260810 07:29:07.842186 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.838478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.24.60.252:35639)
0810 07:29:07.838523 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:07.838525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:07.839355 (+ 830us) server_negotiation.cc:407] Connection header received
0810 07:29:07.839416 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:07.839417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:07.839427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:07.839478 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:07.839595 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.840299 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.841452 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:07.841717 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:07.841751 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:07.841918 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:07.841948 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:07.842024 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:07.842090 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 07:29:07.842110 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:29:07.843431 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18386
I20260810 07:29:07.855707 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18473
I20260810 07:29:07.863035 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18562
2026-08-10T07:29:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2981 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:07 dist-test-slave-vxj2 krb5kdc[18804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:07 dist-test-slave-vxj2 krb5kdc[18804](info): set up 2 sockets
Aug 10 07:29:07 dist-test-slave-vxj2 krb5kdc[18804](info): commencing operation
krb5kdc: starting...
W20260810 07:29:08.982900 24819 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/test-user.keytab.
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.60.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
Entry for principal kudu/127.24.60.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab.
I20260810 07:29:09.105916 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33517
--webserver_interface=127.24.60.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33517,127.24.60.253:36421,127.24.60.252:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:09.188794 18818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:09.189018 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:09.189036 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:09.189056 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:09.190219 18818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:09.190263 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:09.190276 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:09.190289 18818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:09.190301 18818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:09.191726 18818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33517,127.24.60.253:36421,127.24.60.252:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.254.keytab
--principal=kudu/127.24.60.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.254:33517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.60.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:09.191917 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:09.192085 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:29:09.197031 18818 server_base.cc:1034] running on GCE node
W20260810 07:29:09.197000 18825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.197024 18827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.197012 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.197623 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:09.198204 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:09.199427 18818 hybrid_clock.cc:648] HybridClock initialized: now 1786346949199418 us; error 60 us; skew 500 ppm
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:09.206698 18818 init.cc:377] Logged in from keytab as kudu/127.24.60.254@KRBTEST.COM (short username kudu)
I20260810 07:29:09.207127 18818 webserver.cc:460] Webserver started at http://127.24.60.254:45297/ using document root <none> and password file <none>
I20260810 07:29:09.207317 18818 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:09.207357 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:09.207453 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:09.208539 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data/instance:
uuid: "c6a3c5917e484c9caee7def8513bab18"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "336b1264e615cafde7e3759bfe9e58f8"
server_key_iv: "64f6dd81a13e6e325e81c0b78259e561"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.208850 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal/instance:
uuid: "c6a3c5917e484c9caee7def8513bab18"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "336b1264e615cafde7e3759bfe9e58f8"
server_key_iv: "64f6dd81a13e6e325e81c0b78259e561"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.210283 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:29:09.211140 18835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.211342 18818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:09.211452 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal
uuid: "c6a3c5917e484c9caee7def8513bab18"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "336b1264e615cafde7e3759bfe9e58f8"
server_key_iv: "64f6dd81a13e6e325e81c0b78259e561"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.211551 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:09.233060 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:09.235648 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:09.235776 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:09.240314 18818 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.254:33517
I20260810 07:29:09.240362 18894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.254:33517 every 8 connection(s)
I20260810 07:29:09.240736 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:09.241492 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18818
I20260810 07:29:09.241617 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-0/wal/instance
I20260810 07:29:09.241811 24819 external_mini_cluster.cc:1428] Setting key 1941384ecc3fe0d7cdc95fb1d4b472d2
I20260810 07:29:09.241992 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:09.243885 18895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.60.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.253@KRBTEST.COM" created.
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:09.257944 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.244885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.24.60.254:33517)
0810 07:29:09.245052 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.245058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.245097 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:29:09.245716 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.245719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.245743 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.245847 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.246476 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.247901 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.248753 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.249336 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.249458 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.254208 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.254215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.254216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.254223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.256225 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.256639 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.256643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.256643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.256731 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.256978 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.256979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.256980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.257281 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.257372 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.257576 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.257700 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260810 07:29:09.257946 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.244746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33517 (local address 127.0.0.1:57072)
0810 07:29:09.244940 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.245017 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.245540 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.245863 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.245865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.246205 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.246366 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.246370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.248004 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.248004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.248664 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.248665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.249447 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.249448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.249504 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.250061 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.250069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.254063 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.256312 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.256318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.256320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.256530 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.256811 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.256811 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.256812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.256897 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.257351 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.257357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.257421 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.257537 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.257777 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":93,"thread_start_us":88,"threads_started":1}
I20260810 07:29:09.258733 18895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:09.259147 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.258979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54472)
0810 07:29:09.259045 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.259109 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 07:29:09.259315 18836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
I20260810 07:29:09.259785 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.259658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54480)
0810 07:29:09.259721 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.259759 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 07:29:09.259908 18895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:36421: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab.
Entry for principal kudu/127.24.60.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab.
I20260810 07:29:09.270449 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:36421
--webserver_interface=127.24.60.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab
--principal=kudu/127.24.60.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33517,127.24.60.253:36421,127.24.60.252:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:09.309195 18895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } attempt: 1
I20260810 07:29:09.309796 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.309630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54484)
0810 07:29:09.309707 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.309760 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 07:29:09.310191 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.310094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54492)
0810 07:29:09.310141 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.310171 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 07:29:09.310312 18895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:36421: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
W20260810 07:29:09.356408 18904 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:09.356630 18904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:09.356648 18904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:09.356669 18904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:09.357892 18904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:09.357933 18904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:09.357949 18904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:09.357961 18904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:09.357973 18904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:09.359596 18904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33517,127.24.60.253:36421,127.24.60.252:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.253.keytab
--principal=kudu/127.24.60.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.253:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.60.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:09.359894 18904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:09.360188 18904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:09.365319 18910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.365324 18912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.365481 18904 server_base.cc:1034] running on GCE node
W20260810 07:29:09.365391 18909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.365789 18904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:09.366050 18904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:09.367192 18904 hybrid_clock.cc:648] HybridClock initialized: now 1786346949367165 us; error 45 us; skew 500 ppm
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:09.369809 18904 init.cc:377] Logged in from keytab as kudu/127.24.60.253@KRBTEST.COM (short username kudu)
I20260810 07:29:09.370273 18904 webserver.cc:460] Webserver started at http://127.24.60.253:35641/ using document root <none> and password file <none>
I20260810 07:29:09.370579 18904 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:09.370627 18904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:09.370718 18904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:09.371788 18904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data/instance:
uuid: "94e5b6d28a864a1a8aedf027e1096ae2"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "336da78603bbf5347441a32d9136fd7e"
server_key_iv: "7a670acdd366a6e893ba30d08b6dcce7"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.372092 18904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal/instance:
uuid: "94e5b6d28a864a1a8aedf027e1096ae2"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "336da78603bbf5347441a32d9136fd7e"
server_key_iv: "7a670acdd366a6e893ba30d08b6dcce7"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.373492 18904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:29:09.374588 18921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.374775 18904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:09.374876 18904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal
uuid: "94e5b6d28a864a1a8aedf027e1096ae2"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "336da78603bbf5347441a32d9136fd7e"
server_key_iv: "7a670acdd366a6e893ba30d08b6dcce7"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.374961 18904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:09.378489 18895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } attempt: 2
I20260810 07:29:09.378938 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.378807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54508)
0810 07:29:09.378867 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.378907 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 07:29:09.379300 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.379199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54518)
0810 07:29:09.379248 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.379280 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 07:29:09.379397 18895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.253:36421: Network error: Client connection negotiation failed: client connection to 127.24.60.253:36421: connect: Connection refused (error 111)
I20260810 07:29:09.386106 18904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:09.389145 18904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:09.389465 18904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:09.393999 18904 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.253:36421
I20260810 07:29:09.394080 18980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.253:36421 every 8 connection(s)
I20260810 07:29:09.394512 18904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/data/info.pb
I20260810 07:29:09.395624 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:09.395758 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18904
I20260810 07:29:09.395867 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-1/wal/instance
I20260810 07:29:09.396066 24819 external_mini_cluster.cc:1428] Setting key 19478dac2991df1e5e6b8907bb1cd754
I20260810 07:29:09.397397 18981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.60.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.60.252@KRBTEST.COM" created.
I20260810 07:29:09.406775 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.398179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.24.60.254:33517)
0810 07:29:09.398237 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.398239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.398485 (+ 246us) server_negotiation.cc:407] Connection header received
0810 07:29:09.398997 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.398999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.399015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.399081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.399741 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.400253 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.401344 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.401573 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.401664 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.403894 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.403896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.403896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.403904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.405290 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.405730 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.405731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.405732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.405789 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.406048 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.406049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.406050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.406190 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.406265 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.406585 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.406678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:29:09.406975 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.398098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33517 (local address 127.0.0.1:57078)
0810 07:29:09.398363 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.398394 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.398886 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.399157 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.399160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.399416 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.399639 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.399644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.400373 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.400374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.401256 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.401259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.401685 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.401686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.401825 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.402370 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.402379 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.403797 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.405433 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.405440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.405442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.405662 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.405852 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.405852 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.405853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.405970 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.406266 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.406275 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.406407 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.406562 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.406723 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260810 07:29:09.407655 18981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:29:09.415661 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.407943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54532)
0810 07:29:09.408008 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.408022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.408076 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.408294 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.408295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.408462 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.408526 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.408528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.409452 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.409453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.410335 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.410338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.410794 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.410795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.410880 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.411248 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.411255 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.412621 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.414486 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.414487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.414487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.414704 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.414922 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.414923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.414924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.414979 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.415333 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.415334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.415372 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.415434 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.415545 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 07:29:09.415723 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.407990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.24.60.253:36421)
0810 07:29:09.408149 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.408153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.408171 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:29:09.408199 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.408201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.408212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.408272 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.408612 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.409369 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.410418 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.410715 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.410771 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.412706 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.412714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.412714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.412721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.414401 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.414795 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.414798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.414799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.414855 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.415061 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.415062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.415062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.415261 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.415357 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.415474 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.415598 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260810 07:29:09.416457 18981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:09.416831 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.416706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60544)
0810 07:29:09.416755 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.416800 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 07:29:09.416972 18923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
I20260810 07:29:09.417429 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.417344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60550)
0810 07:29:09.417381 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.417408 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260810 07:29:09.417584 18981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:44451: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.60.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab.
Entry for principal kudu/127.24.60.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab.
I20260810 07:29:09.423635 24819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskghm2j2/build/release/bin/kudu
/tmp/dist-test-taskghm2j2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:44451
--webserver_interface=127.24.60.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab
--principal=kudu/127.24.60.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.60.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.60.254:33517,127.24.60.253:36421,127.24.60.252:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:09.466825 18981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } attempt: 1
I20260810 07:29:09.467350 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.467204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60560)
0810 07:29:09.467278 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.467319 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 07:29:09.467783 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.467631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60574)
0810 07:29:09.467701 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.467747 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 07:29:09.467918 18981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:44451: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
I20260810 07:29:09.470546 18895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } attempt: 3
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:29:09.477928 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.470912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.24.60.253:36421)
0810 07:29:09.470950 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.470952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.471007 (+ 55us) server_negotiation.cc:407] Connection header received
0810 07:29:09.471034 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.471035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.471045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.471088 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.471496 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.471985 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.472988 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.473258 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.473312 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.475258 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.475259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.475260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.475267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.476691 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.477078 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.477079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.477080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.477140 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.477377 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.477378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.477378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.477493 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.477570 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.477767 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.477842 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 07:29:09.477929 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.470837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54536)
0810 07:29:09.470887 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.470898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.470941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.471121 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.471122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.471336 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.471418 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.471420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.472065 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.472066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.472896 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.472899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.473328 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.473329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.473411 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.473842 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.473848 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.475168 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.476794 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.476795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.476796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.477022 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.477247 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.477247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.477248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.477313 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.477561 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.477562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.477609 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.477722 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.477829 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 07:29:09.478343 18895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:09.478725 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.478606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60584)
0810 07:29:09.478662 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.478695 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 07:29:09.479053 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.478964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60600)
0810 07:29:09.479009 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.479033 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 07:29:09.479146 18895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:44451: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
W20260810 07:29:09.509760 18988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:09.510035 18988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:09.510071 18988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:09.510118 18988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 07:29:09.510284 18895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } attempt: 1
I20260810 07:29:09.510737 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.510598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60608)
0810 07:29:09.510664 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.510709 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 07:29:09.511099 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.511001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60620)
0810 07:29:09.511049 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.511076 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 07:29:09.511188 18895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:44451: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
W20260810 07:29:09.511544 18988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:09.511588 18988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:09.511605 18988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:09.511647 18988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:09.511695 18988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:09.513309 18988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.60.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.60.254:33517,127.24.60.253:36421,127.24.60.252:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/krb5kdc/kudu_127.24.60.252.keytab
--principal=kudu/127.24.60.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.60.252:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.60.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 06:23:07 UTC on ccycloud.quasar-pwrldj.root.comops.site
build id 4577110
I20260810 07:29:09.513578 18988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:09.513826 18988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:09.519512 18997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.519563 18988 server_base.cc:1034] running on GCE node
W20260810 07:29:09.519531 18993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.519531 18994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.519992 18988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:09.520313 18988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:09.521488 18988 hybrid_clock.cc:648] HybridClock initialized: now 1786346949521492 us; error 45 us; skew 500 ppm
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:09.524114 18988 init.cc:377] Logged in from keytab as kudu/127.24.60.252@KRBTEST.COM (short username kudu)
I20260810 07:29:09.524587 18988 webserver.cc:460] Webserver started at http://127.24.60.252:37913/ using document root <none> and password file <none>
I20260810 07:29:09.524852 18988 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:09.524923 18988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:09.525072 18988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:09.526168 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data/instance:
uuid: "eacc18bc22124126a2cd2ac3f51ad36a"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "0223946211fd97b1844e5df2f77ebbf9"
server_key_iv: "628e16b818620566466e82f8c5e43ffb"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.526579 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal/instance:
uuid: "eacc18bc22124126a2cd2ac3f51ad36a"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "0223946211fd97b1844e5df2f77ebbf9"
server_key_iv: "628e16b818620566466e82f8c5e43ffb"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.528008 18988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 07:29:09.528795 19004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.529093 18988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:29:09.529214 18988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data,/tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal
uuid: "eacc18bc22124126a2cd2ac3f51ad36a"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-vxj2"
server_key: "0223946211fd97b1844e5df2f77ebbf9"
server_key_iv: "628e16b818620566466e82f8c5e43ffb"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.529294 18988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:09.536073 18981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } attempt: 2
I20260810 07:29:09.536530 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.536397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60624)
0810 07:29:09.536458 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.536499 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 07:29:09.536892 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.536791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60630)
0810 07:29:09.536838 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.536867 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 07:29:09.537011 18981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.60.252:44451: Network error: Client connection negotiation failed: client connection to 127.24.60.252:44451: connect: Connection refused (error 111)
I20260810 07:29:09.540482 18988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:09.542949 18988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:09.543118 18988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:09.547181 18988 rpc_server.cc:307] RPC server started. Bound to: 127.24.60.252:44451
I20260810 07:29:09.547232 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.60.252:44451 every 8 connection(s)
I20260810 07:29:09.547603 18988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/data/info.pb
I20260810 07:29:09.548592 19067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:09.548913 24819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskghm2j2/build/release/bin/kudu as pid 18988
I20260810 07:29:09.549031 24819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskghm2j2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346663269703-24819-0/minicluster-data/master-2/wal/instance
I20260810 07:29:09.549299 24819 external_mini_cluster.cc:1428] Setting key 2809be483bd7bd9bae6477d8dd5491d3
I20260810 07:29:09.551360 19067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:09.560587 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.552188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.24.60.254:33517)
0810 07:29:09.552241 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.552243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.552434 (+ 191us) server_negotiation.cc:407] Connection header received
0810 07:29:09.552523 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.552524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.552533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.552587 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.553142 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.553873 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.554889 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.555131 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.555203 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.557619 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.557620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.557620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.557628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.559183 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.559686 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.559686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.559686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.559734 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.559921 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.559921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.559921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.560057 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.560127 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.560451 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.560523 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:29:09.560640 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.549824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.24.60.252:44451)
0810 07:29:09.550005 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.550017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.550049 (+ 32us) server_negotiation.cc:407] Connection header received
0810 07:29:09.550475 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.550481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.550528 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.550642 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.551522 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.552863 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.553955 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.554562 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.554672 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.557063 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.557070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.557071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.557077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.558827 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.559272 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.559275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.559275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.559431 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.559622 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.559623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.559624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.559955 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.560075 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.560227 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.560391 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260810 07:29:09.560640 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.552132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.254:33517 (local address 127.0.0.1:57082)
0810 07:29:09.552336 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.552372 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.552461 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.552623 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.552626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.552986 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.553063 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.553065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.553969 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.553970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.554801 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.554803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.555237 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.555238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.555292 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.555759 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.555767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.557532 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.559265 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.559271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.559273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.559617 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.559798 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.559799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.559799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.559858 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.560131 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.560152 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.560222 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.560313 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.560451 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":30,"thread_start_us":87,"threads_started":1}
I20260810 07:29:09.561311 19067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.254@KRBTEST.COM
I20260810 07:29:09.567979 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.561627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.24.60.253:36421)
0810 07:29:09.561671 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.561673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.561681 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:29:09.561708 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.561709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.561718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.561775 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.562044 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.562511 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.563179 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.563399 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.563486 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.565387 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.565389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.565389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.565398 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.566829 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.567182 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.567184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.567184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.567245 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.567452 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.567453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.567453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.567569 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.567647 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.567818 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.567888 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:29:09.567979 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.561568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.253:36421 (local address 127.0.0.1:54550)
0810 07:29:09.561611 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.561620 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.561654 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.561779 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.561779 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.561920 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.561951 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.561952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.562595 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.562596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.563091 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.563093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.563487 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.563488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.563545 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.563885 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.563889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.565303 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.566931 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.566932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.566933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.567106 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.567322 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.567323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.567323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.567386 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.567640 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.567641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.567689 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.567789 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.567884 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 07:29:09.568318 19067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:09.568989 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.561441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.24.60.254:33517)
0810 07:29:09.561480 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.561482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.561507 (+ 25us) server_negotiation.cc:407] Connection header received
0810 07:29:09.561625 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.561626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.561639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.561671 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.562193 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.562631 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.563301 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.563484 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.563555 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.566190 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.566191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.566191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.566198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.567593 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.567987 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.567988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.567988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.568071 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.568500 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.568501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.568501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.568647 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.568819 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.568850 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.568927 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.252@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.60.253@KRBTEST.COM
I20260810 07:29:09.575354 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.568590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60648)
0810 07:29:09.568632 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.568641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.568676 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.569235 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.569236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.569386 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.569416 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.569417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.570107 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.570108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.570641 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.570643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.571006 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.571007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.571054 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.571443 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.571448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.572703 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.574349 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.574350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.574351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.574518 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.574722 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.574723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.574723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.574777 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.575026 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.575027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.575063 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.575148 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.575249 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 07:29:09.575543 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.568660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.24.60.252:44451)
0810 07:29:09.568999 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.569001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.569010 (+ 9us) server_negotiation.cc:407] Connection header received
0810 07:29:09.569030 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.569031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.569041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.569097 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.569496 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.570020 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.570723 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.570937 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.571008 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.572787 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.572788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.572789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.572795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.574259 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.574602 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.574603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.574603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.574656 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.574848 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.574849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.574849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.574951 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.575023 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.575169 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.575300 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":316}
I20260810 07:29:09.576694 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.569664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.24.60.253:36421)
0810 07:29:09.569707 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.569709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.569717 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:29:09.569740 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.569741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.569750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.569805 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.570273 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.570815 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.571462 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.571667 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.571744 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.573964 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.573966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.573966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.573973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.575388 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.576026 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.576027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.576027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.576076 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.576282 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.576283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.576283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.576375 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.576490 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.576580 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.576637 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 07:29:09.577409 19067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: Bootstrap starting.
I20260810 07:29:09.578351 24819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:09.578521 19067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:09.578814 19067 log.cc:824] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:09.579658 19067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: No bootstrap required, opened a new log
I20260810 07:29:09.579559 18869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c6a3c5917e484c9caee7def8513bab18"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57098
I20260810 07:29:09.581329 19067 raft_consensus.cc:348] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.581454 19067 raft_consensus.cc:729] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacc18bc22124126a2cd2ac3f51ad36a, State: Initialized, Role: FOLLOWER
I20260810 07:29:09.581540 19067 consensus_queue.cc:260] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.581844 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.581918 19067 sys_catalog.cc:564] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:09.581949 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:29:09.583768 19085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:09.583829 19085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:09.586319 18895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } attempt: 2
I20260810 07:29:09.584146 18988 master_runner.cc:186] Error getting master registration for 127.24.60.254:33517: OK, instance_id { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" instance_seqno: 1786346949239153 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:09.587073 18988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.254@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:29:09.593372 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.586621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60662)
0810 07:29:09.586710 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.586721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.586761 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.587012 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.587013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.587213 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.587266 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.587268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.587921 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.587922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.588444 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.588446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.588827 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.588828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.588890 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.589427 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.589432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.590736 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.592243 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.592244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.592244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.592425 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.592642 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.592643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.592643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.592704 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.593009 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.593010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.593058 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.593138 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.593271 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 07:29:09.593391 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.586847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.24.60.252:44451)
0810 07:29:09.586883 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.586884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.586892 (+ 8us) server_negotiation.cc:407] Connection header received
0810 07:29:09.586913 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.586914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.586923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.586970 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.587352 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.587824 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.588557 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.588751 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.588829 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.590828 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.590829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.590830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.590837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.592143 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.592507 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.592509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.592509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.592570 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.592778 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.592779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.592780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.592920 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.592998 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.593219 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.593309 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 07:29:09.595168 18895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18: Bootstrap starting.
I20260810 07:29:09.595839 18895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:09.596149 18895 log.cc:824] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:09.596943 18895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18: No bootstrap required, opened a new log
I20260810 07:29:09.598030 18895 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.598150 18895 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3c5917e484c9caee7def8513bab18, State: Initialized, Role: FOLLOWER
I20260810 07:29:09.598239 18895 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.598574 18895 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:09.598631 18869 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 07:29:09.598704 18869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:09.598582 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.599375 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:09.599520 18869 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.599779 18869 leader_election.cc:288] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [CANDIDATE]: Term 1 election: Requested vote from peers 94e5b6d28a864a1a8aedf027e1096ae2 (127.24.60.253:36421), eacc18bc22124126a2cd2ac3f51ad36a (127.24.60.252:44451)
I20260810 07:29:09.600349 19037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6a3c5917e484c9caee7def8513bab18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eacc18bc22124126a2cd2ac3f51ad36a"
I20260810 07:29:09.600534 19037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [term 0 FOLLOWER]: Advancing to term 1
W20260810 07:29:09.601601 19101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 07:29:09.601621 19037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a3c5917e484c9caee7def8513bab18 in term 1.
W20260810 07:29:09.601655 19101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:09.601854 18837 leader_election.cc:302] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6a3c5917e484c9caee7def8513bab18, eacc18bc22124126a2cd2ac3f51ad36a; no voters:
I20260810 07:29:09.601986 19089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:09.602134 19089 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [term 1 LEADER]: Becoming Leader. State: Replica: c6a3c5917e484c9caee7def8513bab18, State: Running, Role: LEADER
I20260810 07:29:09.602213 19089 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.602608 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a3c5917e484c9caee7def8513bab18. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.602687 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:09.601472 18818 master_runner.cc:186] Error getting master registration for 127.24.60.253:36421: OK, instance_id { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" instance_seqno: 1786346949392781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:09.603415 18818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:29:09.605630 19109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:09.606135 19109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:09.614274 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.605484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.24.60.253:36421)
0810 07:29:09.605542 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.605544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.605902 (+ 358us) server_negotiation.cc:407] Connection header received
0810 07:29:09.606118 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.606119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.606132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.606165 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.607021 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.607509 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.608539 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.609361 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.609432 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.611047 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.611049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.611049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.611057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.612770 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.613210 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.613211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.613211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.613269 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.613538 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.613539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.613540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.613685 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.613758 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.613964 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.614123 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 07:29:09.614840 19037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [term 1 FOLLOWER]: Refusing update from remote peer c6a3c5917e484c9caee7def8513bab18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:09.615404 19089 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:09.617125 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a3c5917e484c9caee7def8513bab18. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.617265 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:09.621824 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.605912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.24.60.254:33517)
0810 07:29:09.605954 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.605956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.614233 (+ 8277us) server_negotiation.cc:407] Connection header received
0810 07:29:09.614333 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.614334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.614346 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.614380 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.615125 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.615812 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.616706 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.616986 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.617036 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.618609 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.618610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.618611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.618619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.620199 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.620644 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.620645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.620645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.620706 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.620918 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.620919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.620919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.621045 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.621164 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.621340 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.621423 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 07:29:09.625645 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.605738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.24.60.252:44451)
0810 07:29:09.605778 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.605780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.613343 (+ 7563us) server_negotiation.cc:407] Connection header received
0810 07:29:09.613456 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.613458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.613472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.613539 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.614783 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.615462 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.616478 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.616788 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.616846 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.622633 (+ 5787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.622634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.622635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.622644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.624329 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.624779 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.624780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.624781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.624858 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.625059 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.625060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.625061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.625220 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.625315 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.625444 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.625538 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 07:29:09.626405 19107 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.626526 19107 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:09.627110 19109 catalog_manager.cc:1347] Generated new cluster ID: c3a4e8111b32423994bafe390424269d
I20260810 07:29:09.627166 19109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:09.627259 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.627348 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:09.627606 19115 mvcc.cc:205] Tried to move back new op lower bound from 7316877105618161664 to 7316877105571332096. Current Snapshot: MvccSnapshot[applied={T|T < 7316877105618161664}]
I20260810 07:29:09.628151 18981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } attempt: 3
Aug 10 07:29:09 dist-test-slave-vxj2 krb5kdc[18804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.60.253@KRBTEST.COM for kudu/127.24.60.252@KRBTEST.COM
I20260810 07:29:09.633533 19109 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:09.634294 19109 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:09.636232 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.628451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.60.252:44451 (local address 127.0.0.1:60672)
0810 07:29:09.628509 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.628519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.628565 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.628734 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.628734 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.628911 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.628961 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.628963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.629828 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.629829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.630656 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.630659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.631129 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.631130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.631213 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.631729 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.631736 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.633328 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.635058 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.635059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.635060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.635291 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.635544 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.635545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.635545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.635622 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.635897 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.635899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.635950 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.636017 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.636120 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 07:29:09.636989 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.628473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.24.60.252:44451)
0810 07:29:09.628525 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.628527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.628619 (+ 92us) server_negotiation.cc:407] Connection header received
0810 07:29:09.628640 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.628641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.628651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.628694 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.629035 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.629756 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.630760 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.631051 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.631117 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.633417 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.633418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.633418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.633426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.634968 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.635369 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.635371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.635371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.635452 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.635683 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.635684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.635684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.635836 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.635904 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.636814 (+ 910us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.636902 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 07:29:09.638566 18981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: Bootstrap starting.
I20260810 07:29:09.639338 18981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:09.639670 18981 log.cc:824] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:09.641515 18981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: No bootstrap required, opened a new log
I20260810 07:29:09.643571 18981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.643729 18981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e5b6d28a864a1a8aedf027e1096ae2, State: Initialized, Role: FOLLOWER
I20260810 07:29:09.643822 18981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } }
I20260810 07:29:09.644197 18981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:09.644258 18953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6a3c5917e484c9caee7def8513bab18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94e5b6d28a864a1a8aedf027e1096ae2"
I20260810 07:29:09.644354 18953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:09.645439 18953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a3c5917e484c9caee7def8513bab18 in term 1.
I20260810 07:29:09.644140 19122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.646013 19122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:09.645711 18952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [term 1 FOLLOWER]: Refusing update from remote peer c6a3c5917e484c9caee7def8513bab18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:09.646811 19107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:09.649559 19136 catalog_manager.cc:1259] Loaded cluster ID: c3a4e8111b32423994bafe390424269d
I20260810 07:29:09.649626 19136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: loading cluster ID for follower catalog manager: success
I20260810 07:29:09.650564 19136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: acquiring CA information for follower catalog manager: success
W20260810 07:29:09.650703 19136 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260810 07:29:09.650775 19122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.650840 19122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:09.651007 19139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a3c5917e484c9caee7def8513bab18. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.651083 19139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:09.654971 19107 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6a3c5917e484c9caee7def8513bab18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3c5917e484c9caee7def8513bab18" member_type: VOTER last_known_addr { host: "127.24.60.254" port: 33517 } } peers { permanent_uuid: "94e5b6d28a864a1a8aedf027e1096ae2" member_type: VOTER last_known_addr { host: "127.24.60.253" port: 36421 } } peers { permanent_uuid: "eacc18bc22124126a2cd2ac3f51ad36a" member_type: VOTER last_known_addr { host: "127.24.60.252" port: 44451 } } }
I20260810 07:29:09.655071 19107 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:09.656064 19109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6a3c5917e484c9caee7def8513bab18: Generated new TSK 0
I20260810 07:29:09.656292 19109 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:29:09.687180 18900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.685238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.24.60.254:33517)
0810 07:29:09.685286 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.685288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.686327 (+ 1039us) server_negotiation.cc:407] Connection header received
0810 07:29:09.686444 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.686445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.686527 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.686564 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.687101 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57112: BlockingRecv error: recv got EOF from 127.0.0.1:57112 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 07:29:09.688158 19070 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.685683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.24.60.252:44451)
0810 07:29:09.685726 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.685729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.687630 (+ 1901us) server_negotiation.cc:407] Connection header received
0810 07:29:09.687729 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.687730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.687741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.687771 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.688095 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60682: BlockingRecv error: recv got EOF from 127.0.0.1:60682 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 07:29:09.689209 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.686040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.24.60.253:36421)
0810 07:29:09.686081 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.686083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.688631 (+ 2548us) server_negotiation.cc:407] Connection header received
0810 07:29:09.688729 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.688731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.688742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.688771 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.689085 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54572: BlockingRecv error: recv got EOF from 127.0.0.1:54572 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 07:29:09.692011 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.691008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.24.60.253:36421)
0810 07:29:09.691046 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.691048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.691266 (+ 218us) server_negotiation.cc:407] Connection header received
0810 07:29:09.691440 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.691441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.691476 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.691544 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.691963 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54580: BlockingRecv error: recv got EOF from 127.0.0.1:54580 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 07:29:09.695197 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.694441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.24.60.253:36421)
0810 07:29:09.694478 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.694480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.694704 (+ 224us) server_negotiation.cc:407] Connection header received
0810 07:29:09.694797 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.694799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.694809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.694837 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.695146 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54594: BlockingRecv error: recv got EOF from 127.0.0.1:54594 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 07:29:09.700354 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.699548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.24.60.253:36421)
0810 07:29:09.699587 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.699589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.699816 (+ 227us) server_negotiation.cc:407] Connection header received
0810 07:29:09.699915 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.699916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.699928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.699958 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.700300 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54608: BlockingRecv error: recv got EOF from 127.0.0.1:54608 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 07:29:09.706956 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.706068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.24.60.253:36421)
0810 07:29:09.706110 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.706112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.706353 (+ 241us) server_negotiation.cc:407] Connection header received
0810 07:29:09.706455 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.706457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.706469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.706525 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.706898 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54612: BlockingRecv error: recv got EOF from 127.0.0.1:54612 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 07:29:09.717391 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.716458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.24.60.253:36421)
0810 07:29:09.716514 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.716516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.716777 (+ 261us) server_negotiation.cc:407] Connection header received
0810 07:29:09.716874 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.716876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.716887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.716916 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.717329 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54618: BlockingRecv error: recv got EOF from 127.0.0.1:54618 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 07:29:09.736040 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.735081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.24.60.253:36421)
0810 07:29:09.735128 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.735130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.735413 (+ 283us) server_negotiation.cc:407] Connection header received
0810 07:29:09.735533 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.735534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.735547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.735580 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.735976 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54624: BlockingRecv error: recv got EOF from 127.0.0.1:54624 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 07:29:09.770787 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.769764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.24.60.253:36421)
0810 07:29:09.769817 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.769819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.770116 (+ 297us) server_negotiation.cc:407] Connection header received
0810 07:29:09.770225 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.770226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.770238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.770268 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.770724 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54632: BlockingRecv error: recv got EOF from 127.0.0.1:54632 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 07:29:09.837540 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.836684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.24.60.253:36421)
0810 07:29:09.836763 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.836766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.836946 (+ 180us) server_negotiation.cc:407] Connection header received
0810 07:29:09.837013 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.837014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.837025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.837068 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.837462 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54648: BlockingRecv error: recv got EOF from 127.0.0.1:54648 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 07:29:09.967583 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:09.966823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.24.60.253:36421)
0810 07:29:09.966899 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.966902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.967010 (+ 108us) server_negotiation.cc:407] Connection header received
0810 07:29:09.967059 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.967060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.967072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.967124 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.967469 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54650: BlockingRecv error: recv got EOF from 127.0.0.1:54650 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 07:29:10.225580 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:10.224907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.24.60.253:36421)
0810 07:29:10.224975 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.224978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.225008 (+ 30us) server_negotiation.cc:407] Connection header received
0810 07:29:10.225035 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.225036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.225046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.225089 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:10.225482 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54652: BlockingRecv error: recv got EOF from 127.0.0.1:54652 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:29:10.584299 19085 catalog_manager.cc:1259] Loaded cluster ID: c3a4e8111b32423994bafe390424269d
I20260810 07:29:10.584386 19085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: loading cluster ID for follower catalog manager: success
I20260810 07:29:10.585187 19085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: acquiring CA information for follower catalog manager: success
I20260810 07:29:10.585439 19085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eacc18bc22124126a2cd2ac3f51ad36a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:10.651345 19136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94e5b6d28a864a1a8aedf027e1096ae2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:10.741559 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.738827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.24.60.253:36421)
0810 07:29:10.738963 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.738965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.739000 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:29:10.739025 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.739026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.739082 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.739151 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:10.739363 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.739868 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.740668 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.740867 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.740921 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.741133 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:10.741300 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:10.741364 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.741459 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.741481 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260810 07:29:10.746070 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.743594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.24.60.252:44451)
0810 07:29:10.743713 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.743715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.743736 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:29:10.743773 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.743774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.743810 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.743872 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:10.743981 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.744535 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.745392 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.745596 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.745653 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.745790 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:10.745894 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:10.745952 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.745987 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.746004 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260810 07:29:10.751255 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.747980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.24.60.253:36421)
0810 07:29:10.748044 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.748046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.748057 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:29:10.748141 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.748142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.748154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.748215 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:10.748414 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.749217 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.750405 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.750707 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.750775 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.750944 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:10.750980 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:10.751030 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.751119 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.751147 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:29:10.751325 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.748104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.24.60.254:33517)
0810 07:29:10.748256 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.748258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.748264 (+ 6us) server_negotiation.cc:407] Connection header received
0810 07:29:10.748304 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.748305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.748343 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.748448 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:10.748678 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.749291 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.750183 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.750436 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.750513 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.750687 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:10.750814 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:10.750880 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.751195 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.751227 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260810 07:29:10.751804 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.748045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.24.60.252:44451)
0810 07:29:10.748216 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.748218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.748332 (+ 114us) server_negotiation.cc:407] Connection header received
0810 07:29:10.748435 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.748436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.748448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.748503 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:10.748638 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.749373 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.750818 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.751141 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.751204 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.751409 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:10.751443 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:10.751476 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.751697 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.751729 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260810 07:29:10.753468 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18818
I20260810 07:29:10.763360 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18904
I20260810 07:29:10.771544 24819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskghm2j2/build/release/bin/kudu with pid 18988
2026-08-10T07:29:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2908 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5890 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287512 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST